Miguel Ventura's blog

A multiplexer for the ever-growing pile of projects

2026-07-12T00:00:00+01:00

My ~/work directory only ever grows. Big projects, medium projects, one-afternoon experiments that never got deleted, clones of other people's code kept around for reference. Twenty-six directories at last count, and I had stopped being able to answer basic questions about my own machine: where did I put that script? What did I even call it? Is it git-tracked, or one rm -rf away from oblivion?

For the rest of this post, let three made-up projects stand in for the whole zoo:

superfoo — the big website. Gets attention almost every day.
midbar — a log-analysis system I had to build once and never cared about again.
tinybaz — a script that parses metadata out of a pile of files. Tiny, but I still reach for it regularly.

I use z (via the oh-my-zsh plugin), and jumping around was never the problem — z sfoo lands me in superfoo just fine. The problems were everything around the jump:

z can only cd. You can't say cp {z baz}/parse.py {z sfoo} — the moment you need a project's path inside a command, you're back to typing it out.
Frecency only helps with projects you visit. The ones you forgot about — midbar and friends — are exactly the ones with no recent rank, and the ones you most need help finding.
Nothing nags you about hygiene, so half the projects end up without git, without a README, or with months of unpushed commits.

I was tempted to build A System™ — a project database, metadata files, an index. I'm glad I didn't, because the whole thing came down to about a hundred lines of zsh built on one principle: the filesystem is the database. Every listing is derived by scanning ~/work at call time, so nothing can ever go stale. The only convention I had to adopt: each project's README starts with a one-line description.

The cake first

Before the recipe, here's what day-to-day feels like with the whole thing in place:

# what did I call that log-analysis thing, and where did I put it?
$ work find logs
midbar                 One-off log analysis for the 2024 gateway migration.

# jump around with fuzzy names, as `z` always allowed...
$ z sfoo

# ...but project names now also work *inside* commands
$ cp ~[baz]/parse_meta.py ~[sfoo]/scripts/

# what's rotting?
$ work doctor
midbar                 no git
superfoo               unpushed 3
tinybaz                no remote
(23 projects clean)

# new projects start life with git, a README and a jump entry
$ work new quuxwatch "Alerting for the quux fleet"
/home/mmv/work/quuxwatch

# retiring a project doesn't break your jump muscle memory
$ work archive midbar
/home/mmv/work/archive/midbar
$ z midb && pwd
/home/mmv/work/archive/midbar

Everything below is the recipe for the above, piece by piece.

zoxide instead of z

zoxide is the actively maintained successor of z, with the same muscle memory. The killer difference is that it's composable: zoxide query sfoo prints the resolved path instead of cd-ing into it. Migrating took two commands:

sudo apt install zoxide
zoxide import --from=z ~/.z    # keep years of frecency habits

Then in .zshrc, replace the z plugin with eval "$(zoxide init zsh)". A trivial wrapper gives you paths in command substitutions:

zq() { zoxide query -- "$@" }

cp $(zq baz)/parse_meta.py $(zq sfoo)/scripts/

`~[name]`: the zsh feature nobody told you about

zsh has dynamic named directories: if you define a zsh_directory_name function, then ~[whatever] anywhere on a command line expands through it. Wire it to zoxide and you get fuzzy project names as first-class path syntax:

zsh_directory_name() {
  [[ $1 == n ]] || return 1
  local dir
  dir=$(zoxide query -- "$2" 2>/dev/null) || return 1
  typeset -ga reply=("$dir")
}

cp ~[baz]/parse_meta.py ~[sfoo]/scripts/
vim ~[midb]/README.md

It tab-completes, it works in any command, and it reads almost like what I wished for in the first place. (If something else in your setup also defines zsh_directory_name, append to the zsh_directory_name_functions array instead of clobbering the function.)

`work`: list, find, doctor, new

The forgetting problem and the hygiene problem are handled by one small script with four verbs, all of them just scans over ~/work:

work new <name> [desc] — mkdir + git init + README with the description + first commit. New projects start compliant, so the convention maintains itself.
work list — every project with the first non-heading line of its README next to it.
work find <pattern> — matches project names and greps all the READMEs. This is the "I know I analyzed those logs somewhere" command that digs up midbar.
work doctor — flags projects with no git, no README, dirty worktrees, unpushed commits. Running it the first time was humbling: 12 of 26 projects weren't git-tracked at all, and one had 506 unpushed commits.

None of these keep any state. Deleting a project directory is removing it from the "database".

Bonus: an archive that zoxide follows

Retiring old projects to ~/work/archive/ keeps the active listing short, but a plain mv breaks your jump history — zoxide still points at the old path. So work archive moves the project and remaps every zoxide entry beneath it:

work-archive() {
  local dir=$WORK_DIR/$1 dest=$WORK_DIR/archive/$1
  [[ -d $dir ]] || return 1
  mkdir -p $WORK_DIR/archive
  # capture the list BEFORE the move: zoxide query --list
  # silently omits paths that no longer exist
  local -a zpaths=(${(f)"$(zoxide query --list)"})
  mv $dir $dest
  local p
  for p in $zpaths; do
    [[ $p == $dir || $p == $dir/* ]] || continue
    zoxide remove "$p"
    zoxide add "$dest${p#$dir}"
  done
}

The comment marks the one real gotcha: query the database before the mv. zoxide hides entries whose directories no longer exist, so if you move first there is nothing left to remap — I found out the hard way. The remap resets frecency scores (there's no "edit" operation, only remove + add), which for archived projects is arguably a feature.

After archiving, z oldproject still lands you in the right place — it's just under archive/ now, and work list no longer mentions it.

Takeaway

I expected this to become a real tool with a real name (it's wmplex in my ~/work, because of course the tool for managing projects is itself a project). It turned out to be a shell script and two config stanzas. If your project directory is drifting out of control, you probably don't need A System either — you need zoxide, one README convention, and a hundred lines of zsh.

GitVersion on NixOS

2026-03-20T00:00:00+00:00

I was recently working with a dotnet project that used GitVersion to stamp builds with version numbers derived from the git history:

<!-- Directory.Packages.props -->
<PackageVersion Include="GitVersion.MsBuild" Version="6.2.0" />

The project also had a few other unusual build dependencies for a dotnet project — nodejs, Java, and others — so I decided to create a flake.nix to be able to quickly spin up a development environment with everything needed.

The problem was that the build started to fail:

.nuget/packages/gitversion.msbuild/6.2.0/tools/GitVersion.MsBuild.targets(26,9): error MSB3073: The command "dotnet --roll-forward Major "~/.nuget/packages/gitversion.msbuild/6.2.0/tools/net8.0/gitversion.dll"" exited with code 139.

Exit code 139 means a segmentation fault. We don't need the full dotnet build to reproduce this — we can run the GitVersion tool directly:

$ dotnet "~/.nuget/packages/gitversion.msbuild/6.2.0/tools/net8.0/gitversion.dll"
[1]    1256082 segmentation fault (core dumped)  dotnet

With help of strace we can see that there's an attempt to load libssl (various versions), and every one of them fails:

$ strace dotnet "~/.nuget/packages/gitversion.msbuild/6.2.0/tools/net8.0/gitversion.dll" 2>&1 | grep ssl
openat(AT_FDCWD, "/nix/store/l0l2ll1lmylczj1ihqn351af2kyp5x19-glibc-2.42-51/lib/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/r7vdyf8pwlqsgd5ydak9fmq3q1i5nm3m-xgcc-15.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/r7vdyf8pwlqsgd5ydak9fmq3q1i5nm3m-xgcc-15.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/r7vdyf8pwlqsgd5ydak9fmq3q1i5nm3m-xgcc-15.2.0-libgcc/lib/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/l0l2ll1lmylczj1ihqn351af2kyp5x19-glibc-2.42-51/lib/libssl.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/r7vdyf8pwlqsgd5ydak9fmq3q1i5nm3m-xgcc-15.2.0-libgcc/lib/libssl.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

The dlopen calls are searching in many places, just not the right ones. GitVersion bundles a native library (git2) that uses dlopen() to load libssl at runtime. On NixOS, libraries aren't in the usual /usr/lib paths, and dlopen doesn't respect RPATH — so the library simply can't find OpenSSL.

The fix is to add openssl to the dev shell and expose it via LD_LIBRARY_PATH in the flake.nix:

{
  devShells.default = pkgs.mkShell {
    # ...

    packages = with pkgs; [
      # ...
      openssl
    ];

    shellHook = ''
      # git2-a418d9d.so (bundled with GitVersion) uses dlopen() to find libssl at runtime.
      # Since dlopen doesn't use RPATH, we must expose openssl via LD_LIBRARY_PATH.
      export LD_LIBRARY_PATH="${pkgs.openssl.out}/lib:$LD_LIBRARY_PATH"
    '';
  };
}

This is a recurring pattern on NixOS: any native dependency that's loaded via dlopen at runtime — rather than linked at build time — won't be found unless you explicitly set LD_LIBRARY_PATH. It's one of those things that "just works" on other distros but requires a bit of manual wiring on Nix.

Dotnet core debugging in Apple Silicon (M1)

2021-08-09T00:00:00+01:00

Apple is transitioning its processors' architecture from Intel (x86_64) to ARM (arm64e) for its line of laptop and desktop computers. The M1 is the first processor in this transition. Programs built for one architecture don't run on the other, but to make this transition smooth, Apple is using a few tricks it already used in past transitions. One of these tricks is Rosetta, which effectively acts as an emulator, allowing x86_64 applications to run in arm64e as if they were running in x86_64. The other trick is fat binaries, which Apple calls universal binaries. These are binaries that contain multiple copies of the code, one for each target architecture. When the time comes to run the program, the system selects the copy of the code that matches the architecture where it's running. The disadvantage is that the binaries get a lot larger, but it's great for distribution as users don't need to worry about downloading the right thing for the architecture they have.

During this transition phase, there are a few apps already using fat binaries, but a lot of apps are still running only on x86_64. At the time this text is being written, this is the case for dotnet core, and for all dotnet core based applications.

Dotnet runs well on arm64 architecture, and has been running on Raspberry Pi, smartphones, AWS Graviton, etc. But on the M1, being macOS, the runtime installation defaults to x86_64.

m1mac$ file `which dotnet`
/usr/local/share/dotnet/dotnet: Mach-O 64-bit executable x86_64

This isn't a problem — Rosetta allows the code to run flawlessly. The problem comes if we attempt to debug this process using LLDB and dotnet-sos. The problems start right after running dotnet sos install. After that command, attempt to run lldb will result in the following errors before the prompt

error: this file does not represent a loadable dylib
error: 'setsymbolserver' is not a valid command.

This means that LLDB is trying to load the SOS plugin, but there's an architecture mismatch — LLDB is running in arm64 and attempting to load a library built for x86_64. Rosetta doesn't help here, because Rosetta only applies to programs being run, not to libraries being loaded into programs that are already running.

We can fix the architecture of the SOS library being loaded, by running dotnet sos install --architecture arm64. This will apparently work (the errors at the beginning of the session will be gone), but no SOS command will work.

(lldb) dbgout
Debug output logging enabled

(lldb) sos ClrStack
LoadLibrary(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.8/libmscordaccore.dylib) FAILED 00000000
Failed to load data access module, 0x80131c4f
You can run the debugger command 'setclrpath <directory>' to control the load of libmscordaccore.dylib.
If that succeeds, the SOS command should work on retry.

From the complaint, it looks like SOS can't locate the CLR runtime libraries, but the problem is actually that we're running LLDB+SOS in arm64 and those CLR libraries are … in x86_64.

Fortunately, there's another workaround. LLDB is installed as a fat binary, which we can easily validate by running

m1mac$ file `which lldb`
/usr/bin/lldb: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/lldb (for architecture x86_64):    Mach-O 64-bit executable x86_64
/usr/bin/lldb (for architecture arm64e):    Mach-O 64-bit executable arm64e

Since we'll be debugging a dotnet process (x86_64) and we'll need to load the dotnet libraries (x86_64), forcing LLDB to start through x86_64 with Rosetta should fix all our issues. This can be done by passing the command through /usr/bin/arch -x86_64, so instead of lldb you would run /usr/bin/arch -x86_64 lldb.

m1mac$ /usr/bin/arch -x86_64 lldb -p 21787
Added Microsoft public symbol server
(lldb) process attach --pid 21787
Process 21787 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff2031bcde libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
->  0x7fff2031bcde <+10>: jae    0x7fff2031bce8            ; <+20>
    0x7fff2031bce0 <+12>: movq   %rax, %rdi
    0x7fff2031bce3 <+15>: jmp    0x7fff20319ad9            ; cerror_nocancel
    0x7fff2031bce8 <+20>: retq
Target 0: (dotnet) stopped.

Executable module set to "/usr/local/share/dotnet/dotnet".
Architecture set to: x86_64-apple-macosx-.

We finally have a working debugger, which we can easily validate by issuing some SOS commands.

(lldb) sos ClrStack
OS Thread Id: 0x358c9 (1)
        Child SP               IP Call Site
0000000304A6B6D8 00007fff2031bcde [HelperMethodFrame: 0000000304a6b6d8] System.Threading.Thread.SleepInternal(Int32)
0000000304A6B820 000000010F3808B7 System.Threading.Thread.Sleep(Int32) [/_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 257]
0000000304A6B830 000000010F81F443 TestProj.Program.D() [~/TestProj/Program.cs @ 32]
0000000304A6B840 000000010F81F3FE TestProj.Program.C() [~/TestProj/Program.cs @ 27]
0000000304A6B850 000000010F81F3BE TestProj.Program.B() [~/TestProj/Program.cs @ 22]
0000000304A6B860 000000010F81F37E TestProj.Program.A() [~/TestProj/Program.cs @ 17]
0000000304A6B870 000000010F815AFB TestProj.Program.Main(System.String[]) [~/TestProj/Program.cs @ 12]

It took me a lot more than I'm proud to admit to get to this point, and it was disappointing that throughout the journey, Googling for most error messages yielded no useful results. Hopefully this post may be found by someone hitting the same problems. At the pace dotnet is evolves, it's likely that this information will soon be obsoleted by a native dotnet arm64e version for macOS.

Advent of Code 2020

2021-01-05T00:00:00+00:00

Another year, another December, another Advent of Code — an advent calendar where each day you get a problem to solve through programming, all wrapped in a surreal Christmas-themed story. @ericwastl does a phenomenal job of creating very varied problems where several types of computer science knowledge can come in handy, but (especially in the age of Google) none are hard requirements to get through the challenges.

There are points for those who finish faster with a public leaderboard to show off your skill. I never competed for time to completion, because my timezone is not ideal (problems are released at 5 AM, when I'm definitely not at the top of my coding game) and because life tends to get in the way of being able to allocate the same time slot every day for the challenges. Not racing for time doesn't mean I don't get a lot of fun. And this year was no exception.

Using fun technology

I write computer code for a living. It's serious business, which means that on the job I often choose boring technology, and you should too. But in AoC you're solving problems with limited scope and you won't have to maintain that code running in customers for years to come. You can chose whatever new technology you heard of yesterday and you can try stuff you have no experience with. Unless you're competing for time (in which case you should use whatever you're most proficient with), it's a great opportunity to gain experience in a new programming language and with new tools and libraries. I've been using the Advent of Code challenges to either learn new programming languages or sharpen my skills in those I use the least.

This year the challenge for me was attempting to solve all problems using purely functional programming with F#. This meant reaching solutions where I didn't mutate any variable, and for that using a functional-friendly but at the same time pragmatic programming langauge. This was the second time I tried solving all problems without mutating a single variable. And this year it felt a lot easier. The only problem where I really felt the need to do mutations was one where the obvious efficient solution for me was using (and mutating) a linked list. I was ready to give up on my goal and started writing a rant for some friends who were also participating. By the time I wrote about it, I remembered an alternative solution¹. A great rubber ducking moment.

Learn with everyone's mistakes (including your own)

This year, a few folks at OutSystems also decided to participate and tried their best to keep the daily pace with the challenges. This wasn't the first time it happened. In previous edition, we'd have a whiteboard at the office with the private leaderboard. Due to the COVID situation none of us is at the office now, which strangely made everything better. Here's how it went:

we created a slack channel specific to advent of code
we pinned a thread with everyone's git repositories
we created a thread for discussions on each particular day
we kept a private leaderboard at AoC site

This meant that like the previous year, we still had the private leaderboard to compare our progress. But the discussion threads and the ability to easily see everyone else's code was the most interesting aspect. In programming there's often no "single right answer" and looking at other people's solutions, especially when they were thinking very differently than you is a great way to expand your way of thinking. In fact this was the way I got to know about AoC in the first place — I was browing Peter Norvig's notebooks when I found a reference to AoC and I got hooked since then.

The daily discussion threads were also great. They enabled us to help eachother keep motivated and they provided a great rubber ducking experince.

Wrap-up

This was probably the year I enjoyed AoC the most. I don't think it was the one with the best problems, but it definitely was the one when I had the most fun. The adding of an additional constraint made it more challenging, but the social aspect was the best. A great thank you to Eric Wastl and to all the friends that walked this journey with me. And in case you're curious, you can look up my solutions here.

A linked list can be implemented with a Dictionary/Map where the value for each key is the next node key. So Immutable Maps can be used for an algorithm that would rely in mutations to a linked list as a substitute data structure. ↩

Three Years of OnePlus One

2017-04-16T00:00:00+01:00

Three Years of OnePlus One

This is the phone everyone wants
but only a few can get.

OnePlus One, the original flagship killer

I used to play Ingress, a geocaching kind of game. It was the predecessor of Pokémon Go. While on vacations, away from my usual territory, I met with a fellow player. We talked, shared stories and exchanged game items. When I got to see the phone he was running the game on, I was impressed. By then I had my second Nexus phone which was signaling the need of retirement. When it was under load, the screen would flicker like old TVs used to do. The performance was already poor for a few apps. The phone would drain its battery fast, even while turned off. This was all quite disappointing, as the phone had about 2 years. Especially disappointing considering the lifespan of the cellphones I had before the smartphone era, and especially disappointing considering that Nexus phones were supposed to be representative of the best Android had to offer.

When I asked my fellow player what phone was he using he introduced me to the OnePlus One. I had no idea what that was about, so he proceeded to explain to me how OnePlus was a recent brand in China, how they were making an awesome device, comparable to flagship products from other brands but much cheaper due to different marketing and distribution techniques. This is the phone everyone wants but only a few can get, he told me. It's invite only. I can hook you up with some folks who might be able to get you an invite. I asked if it was that good and he went on telling me how awesome everything worked in that device. I was certainly impressed. The Net Promoter Score here was clearly over the roof.

Back from vacations I got my hands on an invite and proceeded to order the OnePlus One, also known as OPO, the only device OnePlus had delivered so far.

Along with my device I also got an email message. You are now an ambassador of the OnePlus brand. That's not the same as saying thank you for buying our product. This is a statement that these folks take word of mouth very seriously. This usually means that they put a great deal of effort in customer satisfaction. And the best way to get customers satisfied is by delivering a great product.

Being used to the Nexus devices, and having seen many other Android, iPhone and Lumia devices, the OPO was an incredible surprise. This phone cost about half of other flagship phones. Its camera was incredible. Its performance was incredible. I was used for GPS to take at least 1 or 2 minutes to lock on the Nexus, but on the OPO it was incredibly fast.

But it's not fair to compare a phone that just got into the market with the ones that are already there. Because before installing many apps and filling the storage with lots of photos, many phones feel fast. It's more interesting to compare it after 3 years of usage. And after all this time I can say that the OnePlus One is a truly remarkable device. The apps and games being delivered today on the app store work flawlessly, with great performance. The battery life is still acceptable with nearly 5 days of standby on 2G without internet, or 2 days with WIFI always on, or 1 whole day with WIFI with 4G, which is the same I see recently bought phones performing.

OnePlus One after many falls marking the protective glass

This phone has been with me through hail and heat, snowy mountains and desert sand, climbing and camping. It has seen Europe and Japan. It withstood a few falls which left heavy scars in the screen protector and the case, both bought directly from OnePlus. The phone itself, however, got away without any noticeable scratches, except at the bottom where a tiny scratch can be seen in the metal junction.

With great products come… great problems?

Overall I think the biggest problem that OnePlus might face is having a product that's just too good. It's over 3 years now and I feel no pressure for upgrading. The same happens with other people I know who bought OnePlus One or the newer OnePlus 2 models. Like me, they seem excited to know about what's coming up new with OnePlus devices, but they feel so good with their current device that there's just reason to change. Good, durable quality means OnePlus will have some trouble selling new products to its loyal customers.

Will this be a problem for OnePlus' growth as a company? It's might be, but it's the best kind of problem to have. With this feeling of satisfaction with the product comes a great Net Promoter Score, and that is hugely valuable. How huge? I might not feel the need to upgrade my OnePlus, but since I got mine I influenced 4 persons in my family to switch to OnePlus.

What a great problem to have!

Update: after far too many drops, the bottom of the digitizer finally stopped working correctly in July 2020. The rest of the phone is still working. Six years of flawless lifetime is actually way more than the Nokia 3210 I had (a model famed for being indestructible) resisted on my hands. Can you guess the brand of the new phone I got to replace my One?

Controlled Output through Rx Streams

2016-07-18T00:00:00+01:00

Reactive Extensions make it very easy to create a processing pipeline, especially when some stages on that pipeline are naturally asynchronous operations. This is because just like a Promise (or Task in .NET) can be seen as the asynchronous version of anything that produces a value, the Observable pattern can be seen as the asynchronous version of the Enumerable pattern.

So the following code seems pretty straightforward and easy with Rx:

getListOfLocations()
.flatMap(loc => download(loc))
.map(resource => unzip(resource))
.map(bigText => heavyProcessing(bigText))
.subscribe(registerResult)

We're essentially fetching resources from the internet, unzipping and processing them. The fact that download is asynchronous fits nicely with Rx. But this simple chain might yield a problem: what happens if heavyProcessing becomes the bottleneck?

If heavyProcessing becomes the bottleneck, we'll still be downloading and unzipping content (into memory) but we'll have to queue it up because the processing stage will only observe one item at a time. This means we'll probably run out of memory fast, which makes no sense because we should have full control on the rhythm under which we download the resources or unzip them.

There are some Rx operators that aim at throughput control, like throttle or debounce, but none of them is very useful here, as both end up discarding items when they're observed too fast. In this example we're not interested in discarding any item. We just want to queue up work where we can handle it and then control the rhythm at which we emit items to make sure we won't drown the system under heavy load.

It's still possible to achieve such a solution with Rx by using the following pattern:

Create an observable to represent work tokens.
Use these items to control throughput by either:
- Mapping work tokens into work like
  tokens.map(t => /* disregard t and get next location */)
- Using tokens to buffer up work items where you can
  getListOfLocations().zip(tokens).map(pair => pair.first)
Put a step down the Rx operations chain to emit tokens when resources become available (eg: for each item you process)
Start the whole chain by feeding a few tokens into it (or just one if you're processing one item at a time)

This strategy will only work very well for scenarios where you can control the initial feeding of items into the observable stream, such as controlling the rate at which you read lines from a file you already have in disk. For other scenarios where you have no control over the events that trigger the stream (such as a server taking requests from clients or a user clicking or typing) this strategy can still apply as it enables you to queue items at a stage of the pipeline where they don't need big resources (before turning URLs into downloaded content, or before unzipping that downloaded content into something bigger). Of course, in this latter case, the queue can still grow up too much. But then there's no miraculous solution. To solve such a case, you'll have to consider either discarding work items (through throttling or debouncing) or adding more horsepower into the system.

Note that these controllable observables (for the lack of a better name) are neither hot observables (which emit events despite being subscribed or not) nor cold observables (which only start emitting events once subscribed).

See it in action!

Try in a JS Bin

Conclusion

Reactive Extensions provide a very good way of turning asynchronous processing pipelines into readable code that's easy to reason about. However, when dealing with asynchronous processing, it's always good to take care of understanding how the system's we're building will queue up tasks or attempt to do them all in parallel, and what that will mean regarding system resources. A library such as Rx can be a great help to design a good system, but it isn't in itself an all-in-one solution nor a replacement for thinking about a good architecture.

Going Mechanical: 60 days of Pok3r

2015-10-30T00:00:00+00:00

The case for the Mechanical Keyboard

If you take your craft seriously enough, there's a point in time where the tool becomes the bottleneck. The warriors' sword, the musician's instrument, the writers' pen. As a software engineer, I spend more time thinking than typing, but I still kind of end up using the keyboard all day, every day. However, I never considered the quality of the keyboards I used as a limiting factor to my productivity, and effectively it wasn't... not until I went through two weeks of intense typing durinch which I'd get to the end of the day with aching fingers.

The keyboard I was using was to blame. Cheap, non-mechanical keyboards, have a single membrane over a circuit board that needs to be activated to register the keypress. This usually means that the key has to go all the way down to register. If instead of delivering keypresses you were delivering punches, this would be the equivelent of punching a wall. It would hurt your hand. As-is, it hurts your finger. Probably not enough for you to notice, except after some serious repetition. A mechanical keyboard, on the other hand, has individual switches on every key. By having an individual switch for each key, mechanical keyboards usually register the keypress half-way during the key travel, which means (getting back to the punch metaphor) that you can slow down before hitting the wall and with some training you stop hitting the bottom at all. Trust me, it causes much less strain on you and your fingers.

Mechanical keyboards may be much more expensive than their non-mechanical counterparts, but the investment is totally worth it if you spend a lot of time using your keyboard. In my case, I immediately felt the difference the day I switched keyboards — the aching disappeared and hasn't come back ever since.

There are lots of sites explaining why a mechanical keyboard is worth the price, but I think MechanicalKeyboards.com sums it up pretty well:

Stop and consider how much time you actually spend sitting down (or standing) and typing. Odds are it's a considerable amount. Do you want to spend such a significant portion of your life interacting with an inferior product?

For the price of a nice dinner out, you can purchase an amazing piece of hardware you will use nearly everyday of your life.

Finding your keyboard

Mechanical keyboards are usually much more expensive than their non-mechanical cousins. You can find a pretty good keyboard by 10€ but in the mechanical realm your starting point is much higher, so you should set your starting point expectation at 80€.

There's also a great deal of options regarding mechanical keyboards. The kind of switches you chose, for instance, will influence the kind of noise and key stiffness you'll get, so it'll influence a lot the typing experience. If you're spending 100€+ on a keyboard (80€ was just a reference starting point; chances are you'll spend much more) you want to make sure you're buying something you'll actually like. If you don't have any way of trying a keyboard before buying it, your second best option is to get a switch tester. I got mine from WASD keyboards but UK Keycaps also has a few options.

The switch sampler I bought from WASD Keyboards

The switches in this tester: MX Blue, Brown, Black, Red, Clear and Green

In my case, from looking at the characteristics I had decided on MX Brown switches (no clicky sound, tactile bump on the actuation point, low stiffness), but upon experimenting the switches I ended up deciding on MX Clear because it felt better for me. Only after getting my keyboard with the Clear switches, I had a chance to try another one with Brown switches and verified that I had done the right choice for me. The real point here is that the switch type that feels better is a personal choice, so you should really try them out to see what's best for you.

Another good idea to get acquainted with the world of the mechanical keyboards is to check out the online communities around this matter. I spent some time lurking in /r/mk and checking their wiki. There, besides learning a lot about keyboards, I also got to learn about keyboard enthusiasts. Initially I thought that a keyboard enthusiast would be someone doing intense use of keyboard that would probably have to fit into the category of gamer or programmer. It turns out that keyboard enthusiasts come from all kinds of backgrounds and the only thing in common is really the interest in keyboards and in a superior keyboard usage experience.

Size Matters

Since I was getting a different and rather expensive — for my standards, but not expensive in the mechanical price range — new keyboard, I also wanted to try ditching the numpad. Although the numpad helps a lot when introducing a fair amount of numbers, I actually use it very rarely but I switch a lot between typing and using the mouse, and the numpad is just in the way, forcing a bigger distance for this switch and usually leading to a worse position for the keyboard and a worse posture overall.

However after getting to know more about mechanical keyboards (and keyboards in general) I found out that besides keyboards without numpads (also called tenkeyless or TKL) there are lots of other sizes, and I got especially interested in the 60% size, which are the size of a laptop keyboard. Besides discarding the numpad, 60% keyboards also don't have the arrow keys cluster, further shortening the distance a right hand has to go from a normal writing position to a mouse placed at the right of the keyboard. As a software engineer, I spend quite some time reading code and navigating around, so I use the navigation keys a lot and discarding them completely wouldn't do.

Some 60% keyboards, however, allow you to access those keys either by using a Fn key or by packing them without a cluster of their own (like laptops do). I thought that could be enough for me and decided for the 60%. I regretted this decision every day for the first week of using the keyboard. By the 3rd week, however, I had no regrets. Now I find it strange to use those big, normal keyboards.

The Pok3r

My Pok3r keyboard — MX Clears, White version, UK ISO Layout

The keyboard I settled on buying was a Vortex Pok3r with UK ISO Layout. I would prefer a PT layout but I don't mind what's written in the keys as long as the layout is ISO.

My very first feeling about the Pok3r, right while unboxing, even before trying it, was that this is a very robust keyboard! This was because I was used to plastic keyboards which are quite light, and suddenly I'm holding a keyboard with a solid metal case which feels a lot heavier in comparison. Aluminium is supposed to be light, but this is thick aluminium and it gives a great feeling of robustness.

The typing experience is also great. This is not exclusive from the Pok3r, it has much to do with my choice of switches (the MX Clears feel great for me) and with their choice of keycap material — the slightly rugged surface of the PBT plastic of the keycaps was a pleasant surprise for me; I'm pretty sure that all other keyboards I've used in the last 15 years had ABS keycaps.

The keycaps that come with the Pok3r with their legends for the default behavior and Fn behavior

The Pok3r also has programmable layers that can be switched into. This means that despite not having the F1-F12 keys that are very useful in a lot of programs (most visual debuggers use F8-F12) I could easily program a layer where I replaced the top row keys from 5 all the way to the backspace key with F5-F12. This means I can debug by switching to a layer and pressing single keys instead of having to press combinations like Fn+9 to produce F9.

Besides the great typing experience and its robustness, the Pok3r also looks good. It's so photogenic that it's impossible not to see it while browsing mechanical keyboard communities such as /r/mk.

In Conclusion

Mechanical keyboards may be more expensive, but the extra confort and superior feeling make them totally worth the investment.

Going to a smaller size (like 60%) is not for everyone and might call for an adaptation period if you use a lot the keys you are discarding (probably a non-issue for most laptop users).

The Pok3r is a great product. Looks good and feels good. 100% recommended! Even if it's not the right product for you (due to the layout), if you're a regular keyboard user you should consider other options for mechanical keyboards.

Profile View — you can see the height and slope, as well as a bit of the switches underneath the keycaps

Winning the Hackathon

2015-08-30T00:00:00+01:00

How our mobile app, built in 24 hours, got a 2nd place and what we could've done even better.

In 2013 a bank in Portugal named BPI organized a contest called BPI Appy Day where 400 teams of up to 4 players tried to create a mobile app in 24 hours. The team I ran with produced an app got a nice 2nd place.

My team, MobiLx, presenting our project Fotura

However, at the time I felt like a lot of the teams that were participating were following a bad strategy for this kind of contests. This post is due since then.

Hackathons are seen as programming contents, but some couldn't be further away from that. Programming contests usually have a very limited scope that requires quite a bit of technical ability and the entries are usually code. In a lot of hackathons nowdays, the entries are apps, which is quite different. Sure that there are hackathons very technical in scope, but many of them nowdays aren't much about tech as they are about enterpreneurship.

Winning such competitions isn't as much about devising the most technically awesome solution as it is about selling your app's main idea. Note that this isn't even selling the app but rather selling the idea that it addresses. This isn't much different from pitching any other product in Shark Tank — the details of how the product was made aren't nearly as important as if the opportunity that the product addresses is real.

The rules for selling ideas have a lot of literature of their own, but here's my own summary of rules:

Focus on the idea — remember that's the use-case, not the technical details;
Focus on a single idea — the more you bundle in your pitch, the more confusing everything will get, the harder it will be to sell it;
Make the idea resonate emotionally with the target audience — make sure it's a problem they not only understand but they already felt, so they know it's real.

Selling your work, your ideas and yourself, is probably the most import skill that's absent from the curriculum of most engineering courses. Because we're mostly used to having evaluations focusing exclusively on the technical aspects (at least throughout our academic life), we're quite unprepared for this kind of contests which resemble a lot better the real life.

The timer on the wall — with a tight schedule it's paramount to focus on a polished main use-case instead of a mess of features.

Our project landed in the 2nd place only because in our team of 4 there was 1 element dedicated exclusively to working out the pitch and the presentation of the app, while the other 3 elements were hitting the code as fast and hard as they could. In the end, we had a working app and a pitch that was good, if I may say so. Our app used OCR technology and integration with tax ID databases to ease a lot the work of tracking personal finances and filling tax forms by taking pictures of recipts instead of having to fill in all the info by hand. Our pitch was about how we all dreamed of being rich, and how we envied people who took personal finances seriously and that actually had the self-discipline to go through the work of tracking their expenses and keeping their records tidy. The team that got the 1st place had a far better pitch — their app was about tracking the taking of pills and their pitch started with "I love my grandparents". Bang! Emotional resonance in the jury over the roof.

Is this fair? Shouldn't people who invest a lot in the making of the product (the code) rather than on the pitch and the selling be rewarded by their hard work all the same? Can ideas be worth more than products?

Well, you may still be too used to evaluations exclusively over the technical aspects to accept that it's not the way the real world works. And don't take my word for it or the word of the jury of BPI Appy Day, because in their case the best app as voted by all participants was the very same that got the 1st place from the jury. Even the audience that was made primarily by geeks (who would be more prone to evaluating the technical aspects rather than the pitch) was far from imune to the pitch's seduction.

So before learning the next programming language or framework, how about investing a bit in sharpening that salesman in you?

Celebrating the victory. Might not seem much but it was as much as possible after half a day of work plus 24 hours straight of non-stop development.

Getting to the second place wouldn't be possible without the incredible team I had the priviledge to work with — João Neves, Tiago Simões and Vasco Pessanha

From MySQL to PostgreSQL

2015-05-12T00:00:00+01:00

I recently migrated all my projects from MySQL to Postgresql. These are just some notes to serve as memory aids for me.

I'll keep adding things as I need to recall them.

MySQL	PostgreSQL
show tables	\dt;
use db	\c db
show columns from table	\d table

The Best Programming Language

2013-09-01T00:00:00+01:00

To the kid who asked me about programming languages...

Yesterday, at a barbecue party I met a kid who was very interested in programming. He had been learning about programming languages through the internet and was very interested in the subject and about good practices of programming, the different programming paradigms, etc.. We had a good chat, where his confusion about the amount of programming languages became very obvious — a natural incarnation of the Paradox of Choice. Some of the questions he asked were like:

Which programming language should I learn next?
Should I focus on learning as many languages as possible or just focus on a single language?
Which programming language is the best?

These questions are very interesting to me because they are something that I end up asking myself somewhat often, and I'm pretty confident that they'll appear in the minds of all those who constantly aim at becoming better programmers.

After our long chat, however, I had this epiphany that I want to share with you, kid.

Learning programming languages shouldn't be an end in itself, but rather a means to an end. Just like natural languages: it's rather worthless to learn esperanto if you're not going to use it to communicate something. Communication is the end for each languages serve as the means. Therefore the most important programming language for you should be the one where your ideas can be more easily and elegantly expressed, either to machines or to others. Sometimes this means the better language, other times it means the most commonly known one.

60 days of Kindle Paperwhite

2013-03-04T00:00:00+00:00

TL;DR: It's awesome and I love it!

Amazon had been making fun of me. For a while they sent me promotional emails about the new Kindle Paperwhite. They would let me add it to the shopping cart just to end up telling me that unfortunely some items can't be sent to the shipping address. They knew where I live, so why bother trying to sell me something that they say they can't?

I had waited for the Paperwhite for quite a while. Not because of the back-light, because that actually goes against what I expect the kindle to be: a device that's pretty much book-like. I wanted Paperwhite because I wanted a touch interface instead of bulky keyboards.

The bad

The thing I like the least about the device is that it doesn't allow for the backlight to be completely turned off. There's always a residual light that, at least for me, is always noticeable. The e-ink display technology is awesome because it makes use of environment light. Forcing me to have the backlight always on just doesn't make sense. It also gives a blueish glow to the pages. Fortunately at least this isn't as much of a battery drain as I'd expect: even with the light always on, a single battery charge seems to last forever!

The meh

I've seen people praising the newly added fonts. I've seen people complaining about them not being the best fonts in the world. I didn't even care about changing the font. Did you ever wanted to buy a book but didn't because the font wasn't Scala for serif and Myriad Pro for sans-serif? Of course this is a poor argument: it could still be applied if the device only supported fixed-width fonts. But then, if a book was written in a fixed width font, that could've prevented me from buying it. Well, these fonts don't hurt and are readable enough not to make me care.

Whenever the contents of the screen change, it is possible that remains of the previous contents remain visible on the e-ink displays. A way to get around this is to make the software controlling the screen to do a full refresh (setting the screen all black and all white and then finally printing the new content). Would I make a big deal out of this? No. In almost every book I can see visual evidence of whatever's written in the back of the page that I'm currently reading, so this misbehavior makes the device seem even more book-like to me!

The awesome

Best feature ever: Time until end of chapter. I simply can not overstate how awesome this little estimate is. Until now the estimates were never completely off, and this helps tremendously with scheduling the reading (or the sleeping or whatever else comes next to reading in your agenda).

The Read Later that really works

If you own a Kindle then you know you can email mostly any document to Amazon, to get it delivered on your device. If you use Instapaper and go on browsing the web and marking stuff to read later, you can tell Instapaper to send a bundle of those unread articles to your Kindle. This workflow can be a real productivity booster because it allows you to read the long insightful articles when you have thought time for them, or speed reading (because e-ink displays are awesome for that) the less insightful stuff.

Veredict

Ever since I got myself the Kindle Paperwhite that I've been not only reading more but actually organizing my browsing habits differently, and I really feel more productive with it. Even though 60 days have passed, I'm totally confident that this device will keep bringing me value for quite some time.

Ingress Portal Bursting

2013-01-16T00:00:00+00:00

The world around you is not what it seems.

Or at lease that's how Ingress is presented to the newcommers. The game was created by Google which is by itself quite interesting, and like many Google products, it's been released under an invite-only beta program. It's an ARG where you have to (or at least you're supposed to) actually go to places to perform game actions. Yes, it's a bit like geocaching.

Being a Google closed beta, the game has attracted the attention of a lot of geeks that have been using it as an excuse to go outside and walk. A lot.

Portal attack dynamics

There are many artifacts in the game, but for now let's focus on portals and resonators. Portals are what you want to take over. Resonators are what you place around a portal (up to 8 per portal, one per octant) to make it yours. Conquering a portal usually means destroying the resonators.

Resonators are destroyed by consuming another artifact called a XMP Burster, which come in levels, ranging from 1 to 8. Here's a nice table on damage and range. The amount of damage inflicted by firing a burster varies with the distance to the in-range resonators. Initial research shows a linear decrease of damage with distance, but many say defend that the decrease is rather quadratic or even that it has to do with where you're [pointing at].

Because of this, it's been common knowledge that you should fire the burster while standing on top of the resonator you're trying to fry. Reddit user sikosmurf disagrees with the trend and suggested that the most effective way to take down all resonators on a portal would be to stand in the middle. I decided to do the math myself.

Total damage per position

Let's start by assuming an optimal configuration (all 8 resonators at 35 meters from the portal).

With a linear decrease, the optimal position would effectively be at the center of the portal, for that would be where the sum of the damage over all resonators would be the highest.

If we use a quadratic decrease, the picture changes immediately

However, using a L2 XMP will yield an attack range that makes centering the attack the best option again, even considering the quadratic decay.

Increasing the level of the XMP will only make the pattern more obvious

What if the resonators aren't placed as far from the portal center as they could? In that case, it might make sense to fire even L1 bursters from the portal's center.

L1 against resonators placed in a 25m radius (instead of 35m)

L1 against resonators placed at several distances from the portal, the closest ones at 7m.

These graphs were made using python and matplotlib. You can find the source code here. Any improvement suggestions are welcome.

Image Manipulation with HSL

2012-11-04T00:00:00+00:00

Image manipulation is cool: you can see the result. And people generally like to somehow enhance their pictures. Instagram is now a standard, but before it people were already altering photos to create pop-art like pictures or to enhance their colors. Go check 500px and you'll see that the amount of pictures that were digitally altered is huge.

What's with HSL?

If you've been working with computer software, you must have seen colors being described in terms of RGB values because the additive primary color model is just like what the computer screen ends up using.

But what's a nice model to use for display technology purposes is hardly the most intuitive model for playing around with pictures. Sure that if you've been using image editing software or doing some kind of web development, you've probably grown accustomed to the RGB model and you probably read #FF0000 as "solid red". That's OK.

However, the HSL model is a lot nicer for some image operations. Instead of expressing the colors in 3 dimensions that are about the amount of a primary color, this model represents colors in terms of:

Hue (what you may think of as a pure color)
Saturation (you may think of it as the amount of color) 100% 0%
Lightness (what makes your color closer to black or white) 5% 95%

How do these dimensions help us? Depends on what you're doing. Making a picture gray scale? Just set saturation to 0 everywhere. Want to make it feel lighter or darker? Just play with lightness and leave everything else as is. You can do all that in the RGB model, but it's usually less intuitive.

Even if you feel like you've spent too much time understanding one color model to get interested on another, note that HSL is much easier to learn because it's more related with the way that we, humans, perceive colors.

Image effects that become easy with HSL

To show these effects, let's start with Lenna, because she's a standard.

I'll describe how to use these algorithms using Python with the Python Image Library (PIL) and the colorsys package (which is part of the standard library).

We'll also need a piece of boilerplate code to help us with loading images and storing some relevant attributes (such as dimensions), some functions to help with the conversion from RGB to HSL and back to RGB (because we'll be manipulating the image in HSL but PIL only talks RGB).

from PIL import Image, ImageColor
from colorsys import rgb_to_hls, hls_to_rgb

IMAGE_SRC = 'lenna250.png'

def showme(img):
    """ use PIL standard display """
    img.show()

def h_dist(h1, h2):
    """ distance between color hues in angular space,
    where 1.0 == 0.0 (so distance must wrap around if > 1)"""
    return min(abs(h1+1-h2), abs(h1-h2), abs(h1-1-h2))

def rgb2hls(t):
    """ convert PIL-like RGB tuple (0 .. 255) to colorsys-like
    HSL tuple (0.0 .. 1.0) """
    r,g,b = t
    r /= 255.0
    g /= 255.0
    b /= 255.0
    return rgb_to_hls(r,g,b)

def hls2rgb(t):
    """ convert a colorsys-like HSL tuple (0.0 .. 1.0) to a
    PIL-like RGB tuple (0 .. 255) """
    r,g,b = hls_to_rgb(*t)
    r *= 255
    g *= 255
    b *= 255
    return (int(r),int(g),int(b))

def reload_img():
    global img, sizew, sizeh, maxsize, imgdata
    img = Image.open(IMAGE_SRC)
    sizew, sizeh = img.size
    maxsize = ((sizew/2)**2 + (sizeh/2)**2)**0.5
    imgdata = list(img.getdata())

Now it becomes trivial to do things such as...

Converting Image Color

# change image to single color (hue)
# while keeping saturation and lightness
reload_img()
for hue in range(0,4):
    for i in range(0,len(imgdata)):
        (h,l,s) = rgb2hls(imgdata[i])
        h = hue * 0.25
        imgdata[i] = hls2rgb((h,l,s))

    img.putdata(imgdata)
    showme(img)

And the output is similar to the color palette of a CGA display.

Here we kept the saturation and lightness of colors but based everything into 4 distinct hues in our [0.0 .. 1.0] scale (h = 0.00 , 0.25 , 0.50 , 0.75 ).

For extra impact, we may also increase saturation.

Same as the above (h = 0.75 ) but with s = 1.

Increasing relevance of a single color

We can increase the relevant of a single color by dimming all other colors, and we can easily do that by reducing their saturation. The formula is simple: the more distant a pixel's color hue from our target color hue, the more we'll reduce the saturation.

We can use the above function hdist to get how distant the pixel hue is from ours. Since the hue space goes from 0 to 1 and wraps around, the maximum distance is 0.5. If we double the result from hdist we'll have a number ranging from 0 to 1 that represents the distance to our target color. Having a number between 0 and 1 is great because it means that we can use this number directly as the saturation value or use it as a correction coefficient by multiplying it with the current saturation value.

reload_img()
for i in range(0,len(imgdata)):
    (h,l,s) = rgb2hls(imgdata[i])
    ## Here we chose ONE of the following formulas:
    # dim all but blues
    s *= (1-h_dist(h, 0.75)*2) 
    # dim all but blues (quadratic makes our "close to blue" mean closer)
    s *= (1-h_dist(h, 0.75)*2) ** 2
    ## don't correct, replace
    s = (1-h_dist(h, 0.75)*2) 
    imgdata[i] = hls2rgb((h,l,s))
img.putdata(imgdata)
showme(img)

Linear correction, Quadratic correction and Linear replacement respectively. You should be able to see the original image by hovering the mouse or tapping on any of the images.

As you can see, blues are mostly kept while all other colors fade. This makes them more noticeable.

Movie like effects

We can easily get a Predator-like vision by using lightness directly as hue, with full saturation.

# the predator vision
# full saturation, lightness-based hue, constant lightness
reload_img()
for i in range(0,len(imgdata)):
    (h,l,s) = rgb2hls(imgdata[i])

    s = 1 # one
    h = l # L (lowercase-L, probably the worst variable name)
    l = 0.5
    imgdata[i] = hls2rgb((h,l,s))

img.putdata(imgdata)
showme(img)

We can also simulate a Terminator-like vision by switching everything to red hue, using full saturation, and putting lightness to its extremes — making everything light lighter and everything dark darker.

The trick here is to always use smooth corrections for lightness. If we just did something like

if l < 0.5: l = l * 0.7
else: l = min(l * 1.5, 1)

the resulting image would clearly reveal the lack of smoothness of our approach in areas that should smoothly pass through the threshold

In order to overburn lightness smoothly, we can use a S-curve function such as an adaptation of the Logistic function. Our resulting code could then be something like

# the terminator vision
# red hue, full saturation, lightness based on original lightness
# but S-curved (more gravity at the extremes of the scale)
reload_img()

# we'll use Euler's constant in the Logistic function's formula
e = 2.718281828459045

for i in range(0,len(imgdata)):
    (h,l,s) = rgb2hls(imgdata[i])

    h = 0
    s = 1
    # Adaptation of the Logistic Function
    # A factor of 4+ is OK but the bigger the number, the bigger the gravity
    # towards extremes of the scale.
    s_curve_factor = 10
    l = 1/(1+e**(s_curve_factor-l*s_curve_factor*2))
    imgdata[i] = hls2rgb((h,l,s))

img.putdata(imgdata)
showme(img)

Rainbows

By rotating the hue value, it's trivial to create a rainbow.

# hue according to pixel position, increasing from
# top to bottom
h = float(i)/len(imgdata)

# hue according to pixel position, in a radial progression from
# image center towards corners
h = ((x-sizew/2)**2 + (y-sizeh/2)**2)**0.5 / maxsize

# radial rainbow
h = ((x-sizew/2)**2 + (y-sizeh/2)**2)**0.5 / maxsize
# but instead of replacing original data, we average it with the original
# data with weights (3/4 original, 1/4 rainbow)
imgdata[i] = tuple((3*a+b)/4 for a,b in zip(imgdata[i],hls2rgb((h,l,s))))

By merging the radial rainbow with the original colors, in this particular case, we end up with both a subtle aura and heterochromia iridum.

In Summary

Any of the presented effects is more intuitive to create when operating in an HSL space than when operating in the RGB one. Unfortunately due to the popularity of the RGB notation, especially in web development, people seem to forget that other color systems exist, including ones that may be more appropriate to create whatever effect they need.

Scripting WinDBG with PyKd

2012-09-02T00:00:00+01:00

Why WinDBG?

WinDBG is the debugger for Microsoft Windows. If you don't believe me, just ask any of Microsoft's support escalation engineers. It is extremely powerful but has a very steep learning curve, so usually it's used only when Visual Studio isn't up for the job.

Why Automation?

Often you might find yourself doing repetitive work in the debugger console. For instance iterating over items of a linked list or printing some property of all .NET objects of type X or just chewing up data from some memory region. Sometimes you just want to follow a lot of pointers until you get to the stuff that matters, and that means you'll spend a lot of time doing copy-paste of memory addresses and playing Where's Waldo with cryptic data.

The problem is that most WinDBG commands weren't designed with automation in mind. There are some commands to ease automation... it is easy to iterate over a linked list and there's a .foreach command that's able to parse the output of some other command in a somewhat limited fashion. However the mechanisms that exist are neither easy to use, easy to learn, or even sufficient.

It would be awesome if WinDBG provided some well structured scripting language for aiding with this kind of tasks.

PyKd to the rescue!

PyKd is a WinDBG extension that allows you to integrate Python scripts with WinDBG. Python is awesome for parsing strings and lists, and it's extremely easy to learn and to write. The idea of running Python in WinDBG is so good that I believe Microsoft should integrate this extension with the debugger! (this will probably never happen, but it would be awesome if it did).

Unfortunately PyKd's documentation is at the time mostly in Russian only, but this may very well change in a near future.

Installing and Using

PyKd installation is very straightforward if you use the installer. However I already had Python installed, so I took the zip file only. Due to this I sumbled upon some problems:

The pykd.pyd file must be placed in the winext directory inside your WinDBG installation directory.
PyKd assumes that you have the PYTHONPATH environment variable defined. If you don't, invoking the extension commands will freeze WinDBG (you'll get a BUSY prompt forever).

The usage is very straightforward, especially if all you want to do can be achieved by issuing debugger commands and parsing their output. You can use !pycmd to get a Python shell right in your WinDBG session and dbgClient.dbgCommand to issue debugger commands from python:

.load pykd.pyd
!pycmd
Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>from pykd import dbgClient
>>>loaded_modules = dbgClient.dbgCommand("lm")

Alternatively you can use !py <file> to run a script from <file>.

How cool is that?

It's just awesome! Here are some basic examples of how PyKd can save your bacon:

Listing all Namespaces of WebServiceMethod

You could use !dumpheap -stat WebServiceMethod to get the MT of WebServiceMethod and then !dumpheap -mt <mt> -short to list all WebServiceMethod object addresses. The output of !dumpheap -short can be easily parsed by .foreach, but then you'd have to run !dumpobj to get the address for the namespace property and finally !dumpobj on this address. The output of !dumpobj isn't suitable for .foreach, but it's no trouble for a python script.

import pykd

def dump_soapclientmethod():
    # get all SoapClientMethod's
    soapcliaddrs = pykd.dbgCommand("!dumpheap -mt 0000064283abea38 -short").split("\n")

    print "### found %d soap client addresses" % (len(soapcliaddrs))

    for addr in soapcliaddrs:
        # dumpobj to get object properties
        do = pykd.dbgCommand("!do %(addr)s" % { 'addr': addr })
        # get the line for 'action' property
        actionline = [line for line in do.split('\n') if 'action' in line]
        # line ends with "<address> action" and we want the <address>
        actionaddr = actionline[0].split()[-2]
        # get the string in the retrieved <address>
        doaction = pykd.dbgCommand("!do -nofields %(addr)s" % {'addr': actionaddr}).split("\n")
        print "%s -> %s" % (actionaddr, doaction[-2])

the code above renders an output like

### found 125 soap client addresses
00000001c1755b48 -> String: http://schemas.microsoft.com/sharepoint/soap/List
00000001e2085640 -> String: http://schemas.microsoft.com/sharepoint/soap/Copy
0000000240fb35c8 -> String: http://schemas.microsoft.com/sharepoint/soap/List
00000002419c4158 -> String: http://schemas.microsoft.com/sharepoint/soap/Copy
...

Listing dynamically generated assemblies for XML Serialization

def dump_assemblies():
    addrs = pykd.dbgCommand("!dumpheap -stat -type Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer").split("\n")
    print "### found %d objects" % (len(addrs))

    for laddr in addrs:
        mt = laddr.split()[0]
        mtdump = pykd.dbgCommand("!dumpmt -md " + mt).split("\n")
        modaddr = [line for line in mtdump if line.startswith("Module:")][0].split()[1]
        mddump = pykd.dbgCommand("!dumpmodule " + modaddr)
        assembly = [line for line in mdump if line.startswith("Assembly:")]
        print assembly[0]

Conclusion

With PyKd you can finally script WinDBG in a decent programming language and write some code to automate those repetitive tasks that make you feel you're wasting time rather than debugging.

django-imagekit watermarks

2012-07-28T00:00:00+01:00

I've been playing around implementing watermark processors for django-imagekit. There was an open request for an overlay processor (which is kind of the same thing), and I wanted to explore a bit more PIL and imagekit processors.

So far, I've created processors that take text and images to overlay in the base image. At the time, the code is available in a fork of imagekit.

Examples

Overlaying text

Using TextWatermark we can overlay some text in the following manner

img = Image.open("drave_base.jpg")

title_mark = TextWatermark("Views from Drave",
    font=("/Library/Fonts/Arial Bold.ttf", 16),
    text_color="red",
    position=(20,20),
    opacity=1)

copy_mark = TextWatermark("Photo taken by myself. No copyright.",
    font=("/Library/Fonts/Arial Bold.ttf", 16),
    text_color="#ff0",
    position=(-20,-20),
    opacity=0.5)

img = title_mark.process(img)
img = copy_mark.process(img)

And the resulting img would be

In TextWatermark, all configuration is optional, except for the text to be written.

You see, you can use names for text_color or HTML color notation (or any format supported by PIL.ImageColor). opacity takes a float ranging from 0 to 1. The position parameter accepts tuples containing horizontal and vertical position values. The values can be specified as strings (such as top, left, bottom, right and center), strings containing percentages (such as "25%") or integers. If the percentages or integers are negative, they're measured form the bottom-right instad of the top-left point.

The font parameter may be a path to a TrueType font file or a tuple with a path and a size. Unfortunely there's no font discovery: you have to specify the full path to the font file.

Overlaying images

With ImageWatermark we can overlay one image on another. We'll use the landscape image with this star:

The watermark can be provided to the processor as a str with the path to the image, an Image object or a storage (file-like object).

Here's some sample code on how we could use this processor.

img = Image.open("drave_base.jpg")

# any of these could be used
watermark = "200-star.png"
watermark = Image.open("200-star.png")
watermark = open("200-star.png") # if you use a file-like object
                                 # remember to close your file when done

scaled = ImageWatermark(watermark,
    position=('center', 'center'),
    scale=True,
    opacity=0.2)

repeated = ImageWatermark(watermark,
    position=('right', 'bottom'),
    repeat=True,
    opacity=0.2)

img_scaled = scaled.process(img)
img_repeated = repeated.process(img)

Would result in img_scaled:

And img_repeated:

For performance tuning, it's better to setup the watermark image to be used by path or storage object. Using an Image object will require the Image to be kept in memory forever (which may be a problem depending on the size and amount of images you'll be using). By default, the watermark will be cached using a weak reference, so the image processor doesn't need to constantly fetch it from storage and the image may be collected in case your process gets low on memory.

I hope this code can be of use to you. If you have any issues or requests, please send them through GitHub.

Why is it still Hard to Learn CSS?

2012-07-21T00:00:00+01:00

Lately I've been doing presentations on CSS to colleagues. CSS is continuously evolving, especially with CSS3, but this was a crash-course that only covers the very basics of CSS with some exercises in between, and the whole thing had to fit half a day. The audience was made of people that have been working with CSS for years now, some of them on a daily basis. Making real web sites. With a lot of style! Why would a course of CSS be useful for people that have been using it so much? Why did almost every attendee give positive feedback on the session?

The simple answer is because you can use something for quite some time without really knowing what you're doing. And why didn't they learn the basics in the first place? Because the way CSS is thaught on the internet is fundamentally broken.

CSS Learning with Online Content

I've never read a book on CSS, nor attended to a workshop or course or whatnot. All I've learned about it was by searching and researching within what's available in the internet. So all the information contained in the presentation was actually available in the internet.

CSS information in the internet can mostly be put into one of three categories:

organized reference material: W3C Standards, Mozilla's MDN documentation, Microsoft's MSDN, ...
tutorials of all kinds and sizes
experiments and hacks where people explain tricky techniques to achieve a specific result or where people compare the behaviors browsers have with the behaviors the standards say they should have

And these categories are accessed by different kinds of audiences

organized reference material is used mostly by very experienced users that take time and dedication to thoroughly read such documentation
tutorials are used by everyone else to get starterd and learn everything afterwards... at least until they become very experienced users
experiments and hacks are used by the experienced users as sources of learning and inspiration or by the rest of the users as source material for a trial-and-error approach to solving a specific problem they're experiencing at a particular time

With so much content available for different audiences, it would be natural to expect that learning CSS was an easy task. The reason why this isn't true is because there's virtually no content dedicated to how to solve the problem you need solving. This may seem like a ridiculous idea because there's no way the authors will guess exactly what you need, but it's not that stupid either because there are quite a few problems generic enough for almost everyone to struggle with them, that could have a little lighter answering.

Everyone ends up needing to layout things on a page, which means how to place and position things on a page. But if you look for positioining you end up finding static, absolute, relative and fixed, coordinate systems, and nothing of relevant for what you're trying to accomplish. Could you have guessed that floats were the right section to look for? And even if you could, how would you then make sense of all the information? Would you, just by reading MDN page on floats be ready to abuse this property and start splitting your layout into multiple columns within a centered page?

So what happens to everyone when they attempt at making their header-columns-footer layout? They resort to Google or to tables. If they get into Google, then they'll reach the clever hacks of people who've been fighting with the column problem for years, walking paths of such difficulty that upon reaching the solution they've even called it the Holy Grail. By then it is already too late: even it they're lucky enough to find a well written article (such as A List Apart's), chances are they're already too tired to understand it and just want to copy paste the code. If it works, then the learned lesson will be that CSS is a kind of a dark magic which can be tamed by copying and pasting random code from the internet.

Keeping the Ultimate Goals in mind

People don't usually wake up thinking hey, today I really feel like learning a lot more about floats! They have other things in their mind and that's what we should aim for when doing presentations or writing content on the web, be it for CSS or for anything else. People want to solve some problem they have on the back of their heads. You don't want to know how the chemicals mix themselves to produce the flavor. You just want to have that cake.

As with everything in life, keeping the ultimate goal in mind will help you getting into what really matters. That's why the approach followed to create our little CSS presentation was:

Here's this oversimplified version of things and that's how we want you to see the world from now on (setting a common ground for concepts, terminology, ...)
There's this thing you want to achieve (the goal, real world problem, that could be as generic as layout things in a webpage or as specific as make those round corners work in IE)
- These are the tools you have to do it
- These are the problems we know you'll hit when using this approach
- This is how to overcome those problems
And now that you know how to do it by hand, here's some info on advanced techniques that could help you accelerate that process

At the end of each topic, people would relate with the goal and with the pains, and so they'd all be very happy to know about the tools and the techniques that could make their life a lot easier. They didn't learn what's the normal flow or the box formatting context, but they did learn how to solve their biggest problems and how CSS wasn't all that magic. Perhaps if we had more time we could, only after, explain all those technical details which combined make up the magic.

To be fair, our task was eased by the fact that we knew a lot about our audience. We knew what they'd been struggling with, what they felt that they lacked. We knew what their problems were, and what goals did they have in mind. But again, I'm pretty sure that laying out things on a page is a goal generic enough for everyone starting to learn CSS to be interested in that problem, so there's no excuse for lack of better learning content on the internet.

New Hopes

Better and better content on web technologies is appearing every day on the internet. Initiatives into improving that content and the discoverability of good content have been sprouting all around. I'm expecting that content quality on all HTML, CSS and JavaScript will still mature a lot making it a lot easier to learn these technologies.

CSS3 has some features that may help tackle some of the problems that where extremely difficult to solve with the primitives CSS2.1 offered, and that ended up being the root cause of why solving layouts with CSS wasn't an easy job. For example, if the CSS Grid Layout module matures enough to become the standard of making layouts in HTML, then learning the floating quirks could be left as an exercise to support the legacy browsers of the future.

Beginnings

2012-07-07T00:00:00+01:00

After taking the time to write some articles for the OutSystems' Community, I took some time to evaluate the experiment. The articles that I wrote were about topics I was very familiar with, but even though I forced myself into doing some research to ensure that the quality I was producing was good enough for publishing. Producing quality content was very important because those articles, although written on spare time, could be regarded as products of my work, and I sure like doing good work.

Taking the time to pick your ideas, research them (accepting both refutation and reinforcement) and, last but not least, organizing them in a writeable form, is no news. You've done it in school, writing reports, and in your job, documenting some research. But in such situations we're so focused on the goal that we often don't think about how much did we actually learned, practiced and improved along the process.

Organizing your knowledge in a way that it gets easily explainable to a broad audience just because you want to takes a lot of time and effort, but I strongly believe that it pays off. Explaining you something can make us both better, and for that reason, I'm starting this little blogging experiment, which I hope you enjoy.

Miguel Ventura's blog

A multiplexer for the ever-growing pile of projects

The cake first

zoxide instead of z

~[name]: the zsh feature nobody told you about

work: list, find, doctor, new

Bonus: an archive that zoxide follows

Takeaway

GitVersion on NixOS

Dotnet core debugging in Apple Silicon (M1)

Advent of Code 2020

Using fun technology

Learn with everyone's mistakes (including your own)

Wrap-up

Three Years of OnePlus One

Three Years of OnePlus One

With great products come… great problems?

Controlled Output through Rx Streams

See it in action!

Conclusion

Going Mechanical: 60 days of Pok3r

The case for the Mechanical Keyboard

Finding your keyboard

Size Matters

The Pok3r

In Conclusion

Winning the Hackathon

From MySQL to PostgreSQL

The Best Programming Language

60 days of Kindle Paperwhite

The bad

The meh

The awesome

The Read Later that really works

Veredict

Ingress Portal Bursting

The world around you is not what it seems.

Portal attack dynamics

Total damage per position

Image Manipulation with HSL

What's with HSL?

Image effects that become easy with HSL

Converting Image Color

Increasing relevance of a single color

Movie like effects

Rainbows

In Summary

Scripting WinDBG with PyKd

Why WinDBG?

Why Automation?

PyKd to the rescue!

Installing and Using

How cool is that?

Listing all Namespaces of WebServiceMethod

Listing dynamically generated assemblies for XML Serialization

Conclusion

django-imagekit watermarks

Examples

Overlaying text

Overlaying images

Why is it still Hard to Learn CSS?

CSS Learning with Online Content

Keeping the Ultimate Goals in mind

New Hopes

Beginnings

`~[name]`: the zsh feature nobody told you about

`work`: list, find, doctor, new