Tricks for using Googlemail at work

von: Thorsten Glaser | am: 17.04.2015 | Kommentare: 0

For these who similarily suffer from having to use Googlemail at work. If anyone else has more of these, please do share.

Deactivate the spamfilter

The site admins can do that. Otherwise, you will have work-relevant eMails, for example from your own OTRS system, end up in Spam (where you don’t see it, as their IMAP sucks) and deleted without asking 30 days later. (AIUI, the only way to get eMails actually deleted from Google…)

Do not use their SMTP service

Use your own outgoing MTA. This brings back the, well, not feature but should-have-been-granted-but-Google-doesn’t-do-it-anyway that, when you write to a mailing list, you also get your own messages into your own INBOX.

Calendars…

I have no solutions for this. I stopped using the Googlemail calendars because they didn’t think it a problem that, when I accept an invitation in Kontact (KDEPIM as packaged in Debian sid), the organiser of the calendar item in the sender’s calendar (for which I do not have write permissions) changes to me (so the actual meeting organiser cannot change anything afterwards) and/or calendar items get doubled. I now run a local uw-imapd (forward-ported to sid by means of a binNMU) for sent-mail folders etc. and a local iCalendar directory for calendars.

I have to give you that one

von: Thorsten Glaser | am: 05.03.2015 | Kommentare: 0

After seeing what the Wildfly (formerly JBoss AS) and Liferay combo does to /tmp, and somewhat attempting to fix it, I saw JVM_TMP in the Debian tomcat7 init script and thought, oh no, not another one.

Is that even safe, what they do here, or is that a possibility to instantly pwn?

The net is full of literature for how to obtain temporary files and directories, but there is nothing about how to reliably obtain paths under /tmp or, more generally, directories not just writable for one single user (think the g+w thing that got FusionForge CVE-2013-1423).

The scenario here is: I am root, and I want to start something as another user, and pass it a stable path, such as /tmp/liferay. So I can just mkdir /tmp/liferay || die; chown thatuser /tmp/liferay and, in the “stop” process, rm -rf /tmp/liferay, right? (Of course not. Also, bad example, as the liferay thing can also be started as thatuser, and our devs regularily need to do that, the init script is there just for the admin convenience and reboot-safety. But I still am interested if there is a secure way to achieve this.)

The tomcat7 scenario is “trivial”: on That Other Init System™, it would just get its private /tmp declared in the .service file, and good is, no more hassle. That’s one I have to give you. (No idea if this is actually shipped in jessie. Our production systems run wheezy anyway, so there is not even the slightest bit of temptation. Plus, it would not solve the liferay issue, see above. Still, a point for going into the right direction.)

The idea here is the same. It creates a directory on start and tears it down on stop. If there was nothing to do on start, the init script could just use mktemp -d. Heck, maybe it still should, but it would need to note down, and communicate to the stop instance, the actual name used. What a drag…

This is something I see popping up from time to time. I want to use stable paths for SSH session multiplexing control sockets in my ssh_config(5) file, but have them on tmpfs (Linux) or mfs (BSD) so they get properly removed on reboot. No Unix traditionally has per-user temporary directories that are clean and created after reboot. (Adjusting the paths is trivial once you have them.) Android has it worse, what with not having a world-writable tmp directory, which the shell needs e.g. for here documents; there are two components here, to have a directory the current user can write to, and to know its location. Some fail at the first, some at the second, some at both, and the classic /tmp is not the cure, as we have seen. (But if you ever see mksh erroring out due to lack of write permissions somewhere (including /sqlite_stmt_journals which used to be it) as non-root on Android, or even as root, set TMPDIR to something writable; it’s tracked, so the change gets active immediately.)

tomcat7 log encoding

von: Thorsten Glaser | am: 26.02.2015 | Kommentare: 0

TIL: the encoding of the catalina.out file is dependent on the system locale, using standard Debian wheezy tomcat7 package.

Fix for ‘?’ instead of umlauts in it:

cat >>/etc/default/tomcat7 <<EOF
LC_CTYPE=C.UTF-8
export LC_CTYPE
EOF

My “problem” here is that I have the system locale be the “C” locale, to get predictable behaviour; applications that need it can set a locale by themselves. (Many don’t bother with POSIX locales and use different/separate means of determining especially encoding, but possibly also i18n/l10n. But it seems the POSIX locales are getting more and more used.)

Update: There is also adding -Dfile.encoding=UTF-8 to $JAVA_OPTS which seems to be more promising: no fiddling with locales, no breakage if someone defined LC_ALL already, and it sets precisely what it should set (the encoding) and nothing else (since the encoding does not need to correlate to any locale setting, why should it).

tomcat7 init script is asynchronous

von: Thorsten Glaser | am: | Kommentare: 0

TIL: the init script of tomcat7 in Debian is asynchronous.

For some piece of software, our rollout (install and upgrade) process works like this:

  • service tomcat7 stop
  • rm -rf /var/lib/tomcat7/webapps/appname{,.war}
  • cp newfile.war /var/lib/tomcat7/webapps/appname.war
  • service tomcat7 start # ← here
  • service tomcat7 stop
  • edit some config files under /var/lib/tomcat7/webapps/appname/WEB-INF/
  • service tomcat7 start

The first tomcat7 start “here” is just to unzip the *.war files. For some reason, people like to let tomcat7 do that.

This failed today; there were two webapps. Manually unzipping it also did not work for some reason.

Re-doing it, inserting a sleep 30 after the “here”, made it work.

In a perfect world, initscripts only return when the service is running, so that the next one started in a nice sequential (not parallel!) init or manual start sequence can do what it needs to, assuming the previous command has fully finished.

In this perfect world, those who do wish for faster startup times use a different init system, one that starts things in parallel, for example. Even there, dependencies will wish for the depended-on service to be fully running when they are started; even more so, since the delays between starting things seem to be less for that other init system.

So, this is not about the init system, but about the init script; a change that would be a win-win for users of both init schemes.

Update: Someone already contacted me with feedback: they suggested to wait until the “shutdown port” is listened on by tomcat7. We’ll look at this later. In the meantime, we’re trying to also get rid of the “config (and logs) in webapps/” part…


PS: If someone is interested in an init script (Debian/LSB sysvinit, I made the effort to finally learn that… some months before the other system came) that starts Wildfly (formerly known as JBoss AS) synchronously, waiting until all *.?ar files are fully “deployed” before returning (though with a timeout in case it won’t ever finish), just ask (maybe it will become a dialogue, in which we can improve it together). (We have two versions of it, the more actively maintained one is in a secret internal project though, so I’d have to merge it and ready it for publication though, plus the older one is AGPLv3, the newer one was relicenced to a BSDish licence.)

Java™, logging and the locale

von: Thorsten Glaser | am: 24.02.2015 | Kommentare: 0

A coworker and I debugged a fascinating problem today.

They had a tomcat7 installation with a couple of webapps, and one of the bundled libraries was logging in German. Everything else was logging in English (the webapps themselves, and the things the other bundled libraries did).

We searched around a bit, and eventually found that the wrongly-logging library (something jaxb/jax-ws) was using, after unravelling another few layers of “library bundling another library as convenience copy” (gah, Java!), com.sun.xml.ws.resources.WsservletMessages which contains quite a few com.sun.istack.localization.Localizable members. Looking at the other classes in that package, in particular Localizer, showed that it defaults to the java.util.Locale.getDefault() value for the language.

Which is set from the environment.

Looking at /proc/pid-of-JVM-running-tomcat7/environ showed nothing, “of course”. The system locale was, properly, set to English. (We mostly use en_GB.UTF-8 for better paper sizes and the metric system (unless the person requesting the machine, or the admin creating it, still likes the system to speak German *shudder*), but that one still had en_US.UTF-8.)

Browsing the documentation for java.util.Locale proved more fruitful: it also contains a setDefault method, which sets the new “default” locale… JVM-wide.

Turns out another of the webapps used that for some sort of internal localisation. Clearly, the containment of tomcat7 is incomplete in this case.

Documenting for the larger ’net, in case someone else runs into this. It’s not as if things like this would be showing up in the USA, where the majority of development appears to happen.

Debian/m68k hacking weekend cleanup

von: Thorsten Glaser | am: 18.01.2015 | Kommentare: 0

OK, time to clean up ↳ tarent so people can work again tomorrow.

Not much to clean though (the participants were nice and cleaned up after themselves ☺), so it’s mostly putting stuff back to where it belongs. Oh, and drinking more of the cool Belgian beer Geert (Linux upstream) brought ☻

We were productive, reporting and fixing kernel bugs, fixing hardware, swapping and partitioning discs, upgrading software, getting buildds (mostly Amiga) back to work, trying X11 (kdrive) on a bare metal Atari Falcon (and finding a window manager that works with it), etc. – I hope someone else writes a report; for now we have a photo and a screenshot (made with trusty xwd). Watch the debian-68k mailing list archives for things to come.

I think that, issues with electric cars aside, everyone liked the food places too ;-)

Debian/m68k hacking weekend commencing soonish

von: Thorsten Glaser | am: 16.01.2015 | Kommentare: 0

As I said, I did not certain events that begun with “lea” and end with “ing” prevent me from organising a Debian/m68k hack weekend. Well, that weekend is now.

I’m too unorganised, and I spent too much time in the last few evenings to organise things so I built up a sleep deficit already ☹ and the feedback was slow. (But so are the computers.) And someone I’d have loved to come was hurt and can’t come.

On the plus side, several people I’ve long wanted to meet IRL are coming, either already today or tomorrow. I hope we all will have a lot of fun.

Legal disclaimer: “Debian/m68k” is a port of Debian™ to m68k. It used to be official, but now isn’t. It belongs to debian-ports.org, which may run on DSA hardware, but is not acknowledged by Debian at large, unfortunately. Debian is a registered trademark owned by Software in the Public Interest, Inc.

Tip of the day: prevent iceweasel from mkdir ~/Desktop

von: Thorsten Glaser | am: 05.01.2015 | Kommentare: 0

If you’re a Unix person instead of e.g. a Microsoft® Windows® person, you’ve probably been annoyed by Iceweasel (or Mozilla™ Firefox®) creating a ~/Desktop directory, among others (things like ~/Downloads).

Here’s a quick fix I found somewhere in the ’net:

mkdir -p -m0700 ~/.config
cat >~/.config/user-dirs.dirs <<'EOF'
XDG_DESKTOP_DIR="$HOME/"
XDG_DOCUMENTS_DIR="$HOME/"
XDG_DOWNLOAD_DIR="$HOME/"
XDG_MUSIC_DIR="$HOME/"
XDG_PICTURES_DIR="$HOME/"
XDG_PUBLICSHARE_DIR="$HOME/"
XDG_TEMPLATES_DIR="$HOME/"
XDG_VIDEOS_DIR="$HOME/"
EOF

Upon next start, Iceweasel (and other XDG-compliant applications) will throw stuff into ~/ instead.

Tip of the day: don’t use –purge when cross-grading

von: Thorsten Glaser | am: 12.12.2014 | Kommentare: 0

A surprise to see my box booting up with the default GRUB 2.x menu, followed by “cannot find a working init”.

What happened?

Well, grub:i386 and grub:x32 are distinct packages, so APT helpfully decided to purge the GRUB config. OK. Manual boot menu entry editing later, re-adding “GRUB_DISABLE_SUBMENU=y” and “GRUB_CMDLINE_LINUX=”syscall.x32=y”” to /etc/default/grub, removing “quiet” again from GRUB_CMDLINE_LINUX_DEFAULT, and uncommenting “GRUB_TERMINAL=console”… and don’t forget to “sudo update-grub”. There. This should work.

On the plus side, nvidia-driver:i386 seems to work… but not with boinc-client:x32 (why, again? I swear, its GPU detection has been driving me nuts on >¾ of all systems I installed it on, already!).

On the minus side, I now have to figure out why…

tglase@tglase:~ $ sudo ifup -v tap1
Configuring interface tap1=tap1 (inet)
run-parts –exit-on-error –verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/bridge
run-parts: executing /etc/network/if-pre-up.d/ethtool
ip addr add 192.168.0.3/255.255.255.255 broadcast 192.168.0.3 peer 192.168.0.4 dev tap1 label tap1
Cannot find device “tap1″
Failed to bring up tap1.

… this happens. This used to work before the cktN kernels.

Valid UTF-8 but invalid XML

von: Thorsten Glaser | am: 19.11.2014 | Kommentare: 0

Another PSA: something surprising about XML.

As you might all know, XML must be valid UTF-8 (or UTF-16 (or another encoding supported by the parser, but one which yields valid Unicode codepoints when read and converted)). Some characters, such as the ampersand ‘&’, must be escaped (“&#38;” or “&#x26;”, although “&amp;” may also work, depending on the domain) or put into a CDATA section (“<![CDATA[&]]>”).

A bit surprisingly, a literal backspace character (ASCII 08h, Unicode U+0008) is not allowed in the text. I filed a bugreport against libxml2, asking it to please encode these characters.

A bit more research followed. Surprisingly, there are characters that are not valid in XML “documents” in any way, not even as entities or in CDATA sections. (xmlstarlet, by the way, errors out somewhat nicely for an unescaped literal or entity-escaped backspace, but behaves absolutely hilarious for a literal backspace in a CDATA section.) Basically, XML contains a whitelist for the following Unicode codepoints:

  • U+0009
  • U+000A
  • U+000D
  • U+0020‥U+D7FF
  • U+E000‥U+FFFD
  • U-00010000‥U-0010FFFF

Additionally, a certain number of codepoints is discouraged: U+007F‥U+0084 (IMHO wise), U+0086‥U+009F (also wise, but why allow U+0085?), U+FDD0‥U+FDEF (a bit surprisingly, but consistent with disallowing the backspace character), and the last two codepoints of every plane (U+FFFE and U+FFFF were already disallowed, but U-0001FFFE, U-0001FFFF, …, U-0010FFFF weren’t; this is extremely wise).

The suggestion seems to be to just strip these characters silently from the XML “document”.

I’m a bit miffed about this, as I don’t even use XML directly (I’m extending a PHP “webapplication” that is a SOAP client and talks to a Java™ SOAP-WS) and would expect this to preserve my strings, but, oh my. I’ve forwarded the suggestion to just strip them silently to the libxml2 maintainers in the aforementioned bug report, for now, and may even hack that myself (on customer-paid time). More robust than hacking the PHP thingy to strip them first, anyway – I’ve got no control over the XML after all.

Sharing this so that more people know that not all UTF-8 is valid in XML. Maybe it saves someone else some time. (Now wondering whether to address this in my xhtml_escape shell function. Probably should. Meh.)

« Ältere Einträge |