Archive for the ‘Webtechnology’ Category

No Comments

How to make dmarc-cat read compressed DMARCS reports

Thursday, February 16th, 2023

Dmarc reports are send as attachments by several email-services.

DMARC reports are in the XML format, so of course the reports are compressed before sending.

For example, OUTLOOK is sending them as gzip compressed:

protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.gz

Other email-servers are sending zipped attachments.

Once the report are saved dmarc-cat seems to be able to decompress .zip reports on the fly, but it fails on gz files.

So this works:

dmrac-cat protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml
dmrac-cat protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.zip

But this fails:

dmrac-cat protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.gz

How to make dmarc-cat parse gz dmarc reports?

There is a poorly documented -t switch


dmarc-cat -h

Usage of dmarc-cat:
-D Debug mode
-N Do not resolve IPs
-S string
Sort results (default "\"Count\" \"dsc\"")
-j int
Parallel jobs (default 12)
-t string
File type for stdin mode
-v Verbose mode
-version
Display version

It’s a bit unclear how it should work, but this will do the trick:

dmrac-cat -t .gz protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.gz

And it also miraculously parses zip files, so this works also:

dmrac-cat -t .gz protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.zip

Add dmarc-cat as an alias for dmarc-cat -t .gz

So just add dmarc-cat as an alias in dmrac-cat -t .gz to .bash_aliases and your good to go:

echo "alias dmarc-cat='dmarc-cat -t .gz'" >> ~/.bash_aliases

This will save you some keystrokes the rest of you life!

dmrac-cat protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.gz
dmrac-cat protection.outlook.com!<domain>!<timestamp-start>!<timestamp-end>.xml.zip
No Comments

Trouble updating some packages in Ubuntu 22.04, because they are kept back

Tuesday, January 17th, 2023

Maybe you have seen this message more often then before, while updating your computer manually through the terminal:

The following packages have been kept back:
<package-name>
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Historically that could be the case with the classic commands

sudo apt update && sudo apt upgrade

Because that command is careful (doesn’t update) about packages that can introduce dependency conflicts

But that update command has been superseded by

sudo apt update && sudo apt full-upgrade -y

The latter commands also remove packages to resolve dependency conflicts, and in most cases that is everything you want.

But now upgraded to Ubuntu 22.04 I see the kept back message more and more, even when I did a sudo apt update && sudo apt full-upgrade -y

And as it seems, that is absolutely OK, it’s part of the new phased roll-out mechanism. Packages that can break things are introduced in batches. First only a small percentage of users are getting the new version, so if there are bugs, only a small number of people are hit, and the bugs can be fixed before it is introduced on a bigger scale.

How to check if packages are `phased`

apt-cache policy <package-name>

e.g. gnome-remote-desktop

apt-cache policy gnome-remote-desktop 
gnome-remote-desktop:
Installed: 42.4-0ubuntu1
Candidate: 42.7-0ubuntu1
Version table:
42.7-0ubuntu1 500 (phased 0%)
500 http://nl.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
*** 42.4-0ubuntu1 100
100 /var/lib/dpkg/status
42.0-4ubuntu1 500
500 http://nl.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Just be a little patient. The update will normally come in a few days.

1 Comment

Find the total size of certain files within a directory tree before deleting them

Friday, January 13th, 2023

Normally I use find, as it is installed by default on any Linux computer or server or terminal I worked with, but lately on my desktop I start using `fdfind` more and more.

Why? Its faster and easier to work with than find.

I really like the user-friendly syntax of `fd` AKA `fd-find` or `fdfind` compared to the classic `find` command.

In Ubuntu the program is installed with `sudo apt install fd-find` and executed as `fdfind`

`fd` uses a regex as the standard search pattern.

Time for some examples.

To find all files in a directory tree that have jpg in their name

Very intuitive and concise.

fdfind jpg

To find all jpg files (extension jpg) in a directory tree

Think I need all files with [e]xtension jpg, the command is again very intuitive:

fdfind -e jpg

To delete all jpg files in a directory tree

Think I need all files with [e]xtension jpg then e[x]ecute a command to delete them [rm], the command is very intuitive:

As the normal delete command in bash is `rm`

fdfind -e jpg -x rm

That’s all.

Another interesting thing to know, what disk-space I’m gonna win by deleting all jpg files.

Find the total size of jpg files within a directory tree (wrongly)

Think: I need all files, and then calculate the filespace of all files.

The normal command of getting a total size of several files is use `du -ch *.jpg` This will list of files and Count a total on the last line. To get just the last line. pipe it to tail, to gets just the last line.

du -ch *.jpg |  tail -1

But du doesn’t work recursive in subdirectory. You can use a trick with globstar, but much easier is it to combine with fd, so you would come to something like this.

fdfind -e jpg -x du -ch | tail -1

But that doesn’t work right, it seems to computes totals for every file, and just show to size of the last result.

Find the total size of jpg files within a directory tree (correctly)

We need the `-X` option here the `execute-batch` command, that runs the command only once on all search results as arguments

fdfind -e jpg -X du -ch | tail -1

Find correctly find the total size of jpg files in a directory and the first level of subdirectories

And with `fdfind` command it’s easy to control Depth, just add a -d option. This will only search in the main and the first subdirectory level.

fdfind -d 2 -e jpg -X du -ch | tail -1

And now you ask yourself. Can I find the size of all jpg files in the third level of subdirectories.

Of course! And easier than you think

Find the total size of jpg files in the third level of subdirectories depth

fdfind --min-depth 4 --max-depth 4 -e jpg -X du -ch | tail -1

See more:

https://github.com/sharkdp/fd

No Comments

How to check if you’re running Wayland on Linux

Friday, January 6th, 2023

Wayland is the new display server protocol used by modern Linux installations. It replaces the old X11 protocol. It’s shipping on Ubuntu 22.04 by default.

To check: open a terminal, and echo the `XDG_SESSION_TYPE` variable.

echo $XDG_SESSION_TYPE

It will output `wayland` or `x11`.

You can still try x11 if you like by selecting that option on login.

UPDATE

This doesn’t work over SSH, it will output: tty

So for that, use this:

loginctl show-session $(loginctl | grep $(whoami) | awk '{print $1}') -p Type | grep wayland

In case Wayland is used it should output:

Type=wayland

See more unix.stackexchange

No Comments

Firefox and Wayland support on Ubuntu snaps and the user-agent

Friday, January 6th, 2023

Ubuntu 22.04 is shipping with Wayland as the default communication protocol for the display server, replacing the old and X11 (X Window System).

Interestingly although Firefox is supporting Wayland natively, the default stable Firefox snap package doesn’t use it. I is still using XWayland as compatibility layer.

How to check if Firefox is using Wayland or X11?

Open:

about:support

and search for `Window protocol`

Window Protocol xwayland

That is intentional, see

https://bugzilla.mozilla.org/show_bug.cgi?id=1631462#c21

So what about Firefox Beta? To install Firefox Beta snap next to Firefox stable, see the earlier blog-post Install seperate Fiefox Beta snap.

Open

about:support

And search for `Window protocol`

Window Protocol wayland

Note the missing X, that means Wayland is used as the communication protocol.

Why doesn’t show Wayland in the User-Agent header of Linux browsers?

To check the the User-Agent in Firefox Beta, navigate to something like a ip-address checker:

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0

Although X11 is not used, it still shows X11. I wonder if that is intentional? Probably that is. User-Agent strings are hampered by historical mistakes, like browser-sniffing.

And setting a detailed User-Agent is also a privacy and security risk. Giving to much and unnecessary information about your system. So it is about limiting exposure to browser-fingerprinting.

Be aware this sucks, once you’re aware there is market-power in user-agents. The major players make the rules of the game.

No Comments

How to check your CPU is vulnerable for Retbleed?

Friday, December 16th, 2022

On Linux checking for known vulnerabilities is quite easy.

grep -r . /sys/devices/system/cpu/vulnerabilities

On a Zen2 processor  you’ll get these results:


/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Retpolines, IBPB: conditional, STIBP: always-on, RSB filling, PBRSB-eIBRS: Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/retbleed:Mitigation: untrained return thunk; SMT enabled with STIBP protection
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected

On a Zen3 processor you’ll get these results:

/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Retpolines, IBPB: conditional, IBRS_FW, STIBP: always-on, RSB filling, PBRSB-eIBRS: Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected

As you can see a Zen2 (Ryzen <5000 series) is vulnerable for Retbleed, why the newer generations are not (Ryzen >=5000 series).