Vitaly Samigullin's blog

emacs tramp: ssh + sudo

2024-01-06T18:58:00+01:00

TIL I can use Emacs Tramp not only to use sudo on localhost with the Emacs running under unprivilidged user:

C-x C-f /sudo:localhost:/path/to/file RET

or that I can ssh with Emacs:

C-x C-f /ssh:pilosus@raspberrypi:/path/to/file RET

but I also can pipeline ssh session with sudo:

C-x C-f /ssh:pilosus@raspberrypi|sudo:raspberrypi:/path/to/file RET

which is a shortcut for:

C-x C-f /ssh:pilosus@raspberrypi|sudo:root@raspberrypi:/path/to/file RET

Unmaintained technical documentation

2024-01-06T18:04:00+01:00

I think that the problem of unmaintained technical documentation has nothing to do with the “wrong” tools. It’s not about what’s better: Notion, Wiki or a bunch of version-controlled Markdown files. Rather, the problem boils down to the following problems:

1. Wrong ownership of the documentation (who really cares: a developer, a tech lead, a product manager, etc.)

2. Wrong documentation form (Diátaxis approach with tutorial, how-to, reference, explanation formats of docs)

3. An attempt to solve a problem with the docs where code or automation can be written or a work process introduced

Ownership & docs form

These two points are better explained with the examples.

Example 1: as a developer of a closed-source piece of software with no public APIs, I probably don’t care about writing the docs in the format of reference. This is because I am responsible for the code, I work with the code on a daily basis and hence I try to write my code to be self-documenting. On the other hand, if today I am an on-duty engineer, I would thank my colleagues for writing a short how-to guide that will help me to quickly fix a rare problem that occurs, say, once a month or a quarter. I’ll likely to write or update such a how-to guide myself until we have capacities or increased problem occurrence to automate the how-to guide, so that it’s the code again and docs are no longer needed. I don’t expect that how-to guides are written by a team lead or a technical writer; it’s developers who need them.

Example 2: as a team lead or a product owner, it’s very likely that I want to have some higher-level understanding-oriented form of docs maintained. So that it’s easier for me to onboard a new developer or explain a a complex concept with its context to someone, etc. I don’t expect devs to write these docs for me, it’s my responsibility and ownership. I am interested in maintaining it and I’ll suffer if the docs are unmaintained when I need to answer the same questions over and over again.

Docs instead of automation or a process

It’s nice to come up with a checklist document for a new employee onboarding:

add the GitHub account to the organisation
add relevant permissions on GCP
send a VPN certificate
…

It’s much better than not having it at all. But it’s even better if you implement the checklist as a automated process with your Infrastructure-as-a-Code approach. It will take more time to implement first, but once done it’s guaranteed to be maintained.

Same with the managerial work processes. It’s nice to outline it with a document first, but once implemented, keep only relevant forms of the docs maintained by the people who are interested to have these docs updated. They still may need help organising the process of the docs maintenance: e.g. every time when someone disturbs a colleague asking how to do something, they are instructed, but also asked to update the docs with the answer they just got. They can also have a task to update the docs during their on-duty rotation.

Multiple WiFi network connections on Rapspberry Pi 5

2024-01-06T16:10:00+01:00

This post was intended to be an answer to this question. But given that I’m a new Raspberry Pi’s StackExchange user and I don’t have enough karma, I cannot post my answers yet, hence a post in my own blog.

Problem statement: I want to have multiple networks to connect my Raspberry Pi 5 device to, e.g. a WiFi connection at home and another WiFi connection at work. Previously, all the WiFI network configuration used to be done with the wpa_applicant alongs with many other network-related tools. Now, it seems, NetworkManager and its plugins took over many Linux distributions, including Raspberry Pi OS.

According to the changelog, in September 2022 and later in December 2023 NetworkManager started to be used as a high-level interface for all the network configuration on Raspberry Pi OS. So, as of January 2024 for Raspberry Pi 5 under Debian 12.4 Bookwork (Raspberry Pi OS) this is how one can have multiple WiFi connection profiles set up.

Headless set-up

Connect to your device and become a root user

$ ssh user@raspberrypi.local
$ sudo -s

Find all the WiFi networks available and network devices

$ nmcli dev wifi
$ nmcli dev

note the SSID (your network ID) and the Device name (e.g. wlan0 of type wifi).

Run a NetworkManager config tool with the text-based interface

$ nmtui

Edit connection -> Add -> WiFi.

Add a new connection profile:

profile name
SSID from the step 2
Security (usually WPA & WPA2 Personal for the home WiFi routers + password)
Other settings if needed (e.g. if no DHCP is used)
<OK>

Restart the NetworkManager service

$ systemctl restart NetworkManager

To switch between the connection profiles manually

$ nmtui

Activate a connection -> Choose a profile needed

The same can be done via the CLI tool

$ nmcli connection down 'profile1'
$ nmcli connection up 'profile2'

Beware not to deactivate your only active connection! Otherwise you may found your ssh session hang. You’ll need to restart the device (if no other connections exist) and reconnect in this case.

Desktop

I haven’t used Debian Bookworm with the GNOME desktop environment, but I guess it uses nm-connection-manager as a GUI alternative to nmcli and nmtui and is a part of the network-manager-gnome package.

More details on NetworkManager:

DNS sinkhole with dnsmasq on local Linux machine

2024-01-01T21:38:00+01:00

Recently I’ve posted a wrap-up on my new tiny DNS analyzer tool called dnseen. One of the development goals was generating a top-list of websites that I visit along with the frequencies, so that I can blacklist websites that I think I spend too much time on without long-term benefits. These are mostly news and social media that use manipulative algorithms to make you a “dophamine addict”. Another thing that I wanted to implement is blacklisting for ads/tracking services that works on the DNS level, so that I don’t need to worry about browser extensions in case I want to try a new browser.

I used to have a list of blacklisted domains in my /etc/hosts file for a while. One obvious drawback for me is the need to lump everything in one file. Instead, in a perfect solution, I’d like to split blacklisted domain into multiple files, so that I can easily break them down by topic and/or automate some hosts file generation from third-party curated lists, like hosts files from Steven Black.

One of the popular options is to set up a Pi-hole on the local network and use it as the local DNS server. I do like this option too, but before picking up ready to use products, I’d like to “reinvent the wheel” and build by own prototype first. This is how I came up with the idea of building my own DNS sinkhole with the help of dnsmasq as a DNS server on my local Linux machine.

Linux + NetworkManager + dnsmasq

After installing dnsmasq on the Fedora 39 Linux machine:

sudo dnf -y install dnsmasq

we need to make sure it’s configured as a NetworkManager plugin rather than systemd, so that dnsmasq daemon is run correctly and redefined the /etc/resolv.conf, the resolver configuration file used for DNS lookups.

First of all, we need to activate the NetworkManager plugin:

$ cat /etc/NetworkManager/conf.d/00-use-dnsmasq.conf
[main]
dns=dnsmasq

Now, we can configure the dnsmasq itself:

$ cat /etc/NetworkManager/dnsmasq.d/00-addn-hosts.conf
# NB! When making changes, don't forget restart dnsmasq & NetworkManager.service:
# $ killall dnsmasq
# $ systemctl restart NetworkManager

# No need to read /etc/resolv.conf as it contains the localhost
# addresses of dnsmasq itself
no-resolv

# Upstream DNS servers
# My router, Google, CloudFlare
server=192.168.0.1
server=8.8.8.8
server=1.1.1.1

# Add debug logging
# $ journalctl --follow -u NetworkManager
log-queries

# Ignore /etc/hosts
no-hosts

# Add custom hosts files;
# If path is a directory, all files from it are loaded
addn-hosts=/etc/hosts.d

So, basically we want to keep our multiple hosts files under addn-hosts=/etc/hosts.d directory and make sure that if a hostname cannot be resolved using these files we fallback to the upstream DNS servers that are the one my local network’s router provides, as well as public ones from Google and CloudFlare.

Now, given that my Fedora machine also runs systemd-resolved, a system service that provides network name resolution to local applications, we want to prevent both dnsmasq and systemd-resolved to mess up with the /etc/resolv.conf file. So we stop and disable systemd-resolved:

$ sudo systemctl stop systemd-resolved.service
$ sudo systemctl disable systemd-resolved.service

We can add a hosts file for testing purposes:

$ cat /etc/hosts.d/news
0.0.0.0 example.com
0.0.0.0 news.ycombinator.com

Next, let’s validate dnqmasq config and restart the relevant services:

$ dnsmasq --test
$ sudo killall dnsmasq
$ sudo systemctl restart NetworkManager

Check if local DNS works as expected

First of all, let’s look at the dnsmasq logs:

$ journalctl --follow -u NetworkManager
...
Jan 01 21:03:16 fedora dnsmasq[16321]: using nameserver 192.168.0.1#53
Jan 01 21:03:16 fedora dnsmasq[16321]: using nameserver 8.8.8.8#53
Jan 01 21:03:16 fedora dnsmasq[16321]: using nameserver 1.1.1.1#53
Jan 01 21:03:16 fedora dnsmasq[16321]: read /etc/hosts.d/news - 2 names

Let’s also make sure /etc/resolv.conf is controlled by NetworkManager:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1
options edns0 trust-ad

Now, let’s see if DNS lookups work properly:

$ dig news.ycombinator.com

; <<>> DiG 9.18.20 <<>> news.ycombinator.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15461
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;news.ycombinator.com.                IN      A

;; ANSWER SECTION:
news.ycombinator.com. 0       IN      A       0.0.0.0

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Mon Jan 01 21:18:57 CET 2024
;; MSG SIZE  rcvd: 65

So, yes, SERVER: 127.0.0.1#53(127.0.0.1) (UDP) is telling us a localhost DNS is being used, and A 0.0.0.0 from the ANSWER SECTION shows us it’s resolved to the 0.0.0.0 IP address from our hosts file.

One the same time, if you try to access the same domain in a web browser or using curl, you will see it’s still accessible. The thing is dig tool is using /etc/resolv.conf settings for DNS, while other applications rely on the DNS settings from the network connection. Let’s see what are these DNS setings using NetworkManager CLI tool:

$ nmcli device show | grep -i dns
IP4.DNS[1]:                             192.168.0.1
IP6.DNS[1]:                             2a02:8383:d:c::1000
IP6.DNS[2]:                             2a02:8383:d:c::1

These are actually DNS addresses from my WiFi router. Given that my machine receives IP-address from the router’s DHCP server, it also gets the DNS settings.

Configuring connection’s settings

We can override the settings in NetworkManager GUI. Open up nm-connection-editor (or Settings -> WiFi -> Network you’re connected to Settings), then IPv4:

Method -> Automatic (DHCP) addresses only
DNS Server -> 127.0.0.1 (instead of automatic one)

then for IPv6:

Method -> Automatic, DHCP only (watch out, it’s different than for IPv4)

This corresponds to the changes in /etc/NetworkManager/system-connections/YOUR-CONNECTION-NAME.nmconnection:

[ipv4]
dns=127.0.0.1;
ignore-auto-dns=true
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns=::1;
ignore-auto-dns=true
method=dhcp

Now restart the NetworkManager and check DNS settings:

$ sudo systemctl restart NetworkManager
$ nmcli device show | grep -i dns
IP4.DNS[1]:                             127.0.0.1
IP6.DNS[1]:                             ::1

All good! Local DNS sinkhole works. Now hosts files under /etc/hosts.d can be orginized easily and will be used for domains resolution in all the requests coming from the configured connection.

P.S. I’m deeply grateful to the Stackexchange community for the help with the dnsmasq config troubleshooting.

dnseen: simple DNS queries analyzer

2023-12-29T20:00:00+01:00

dnseen is my new tiny open source project to ease collection of DNS requests coming from the local machine and aggregating statistics over given period of time. dnseen is written in Clojure programming language (with babashka) and works as a wrapper over tcpdump.

It’s written with the following goals in mind:

Goals

Get a top-list of websites that I visit intentionally

This is mostly to reduce media consumption and remove distracting websites that make you a dophamine addict and bring so little (yes, HackerNews is one of them!). For now I simply add domains I want to block to my /etc/hosts like this:

0.0.0.0 domain

But as my next step I’d like to use dnsmasq and maintain an extended collection of hosts files, possibly using Steven Blacks hosts files collection (upd. see my dnsmasq guide).

Find anomalies related to tracking activities

Even the software that claims to protect user’s privacy may track too much. So I want to collect DNS requests over longer period of time to see if I can find any suspicious activity.

Alternatives

Sure enough, one can use tcpdump, wireshark or other packet analyzer to do the job. But they are relatively complex and designed for troubleshooting rather than long-term analytics based on aggregated data.

Or one can use tools developed to track DNS queries specifically, like dnstop does. It’s nice, but again, works more like top tool showing queries happening at the moment rather than trends.

I needed something that allows me to collect data over longer time windows, apply filters and get some analysis done on the aggregated results.

Update 2024-01-07: the closest alternative is probably Pi-hole. It’s rich in features and visualisations, but probably a bit too “intrusive” to be run on my local machine: too much code that I don’t have time to investigate, webserver with php website for the stats dashboard. I’d prefer to use it on a dedicated machine (like a RaspberryPi device) if needed.

dnseen

dnseen is a simple DNS queries analyzer that works on top of the tcpdump logs. For now it works on Linux only (tested with Fedora 39), and get use of the systemd. Under the hood it’s just a systemd service running tcpdump and writing simple plain-text log to the file system. The analyzer program parses logs, applies filters from command line options, and prints the statistics report to the terminal. Reports can be printed for humans in pretty tabular format or as a plain text to be further used in the Unix pipelines. The program also comes with the installer script to make the installation quick and easy.

Findings

So far dnseen seems to be handy. When it comes to finding sources of excessive media consumption, it allowed me to block some news and social websites that I don’t really want to spend my time with.

Regarding detecting tracking activities, Fedora’s NetworkManager turned out to send regular requests to help with the captive portal detection that I don’t need on my home desktop at all. I also blocked quite a few Mozilla’s domains related to the services I never allowed Firefox to use, like Mozilla Location Service.

Conclusion

All in all, the project seems to be useful. tcpdump logs that systemd service’s collecting can also be used for further analysis with other tools, e.g. to determine DNS queries per domain with the breakdown by hours.

Also, doing data analysis with Clojure is such a pleasure! Threading macros is a game changer as it brings the power of Unix pipelines to the functional programming language.

Using babashka feels good too. It’s not my first babashka project, but the first one using babashka.cli for command-line options parsing. It’s simple, yet powerful!

Developing with Clojure and Babashka allow me to work in a flow, with almost no distractions to external world, as the language and its ecosystem are very well designed, self-sufficient, and come with nice tooling like Emacs CIDER. It feels like tinkering in the garage in my childhood!

Recording a screencast on Linux

2023-12-21T23:55:00+01:00

TIL I can record a screencast using GNOME’s tool, trim it with ffmpeg like this:

cd ~/Screencasts
ffmpeg -i screencast.webm -ss 00:00:03.120 -t 00:00:11.500 output.webm

then resize with ImageMagick:

magick output.webm -resize 50% -layers coalesce output.webp

and simply use webp as is instead of converting it to gif, as browser support for webp is pretty decent nowadays.

My Linux journey so far: status update 2023

2023-12-10T13:13:00+01:00

I started using Linux at the high school in 2002. 16 years old at that time, I didn’t have any contacts with the other Linux users in the town where I lived. I did have a slow occassional dial-up internet connection, a few books available in the local book store and a mail order shop to purchase CDs with various Linux or BSD distributions. It all started with the Red Hat Linux 7.3 Valhalla back then. I also tried Slackware, Debian, FreeBSD 4.5 and whatnot on my desktop.

During my time at the university (2003-2008), Linux was a no-go: the whole education process was Windows-centric. After trying some workarounds with Wine, I realised I was wasting more time on the “motorcycle maintenance” than on the ride itself, so I gave up.

In 2011 I got Arch Linux distro installed on my pretty low-performant machine I used for pet projects and other non-work-related things. It was all the way the hacker machine, for sure! Tiling windows manager (i3-wm and even ratpoison), even more never-ending “motorcycle maintenance” with Emacs and OS customizing!

I also had Gentoo Linux on my second, even less performant tiny machine. You can imaging building the Firefox browser on it from the source code, let alone upgrading the whole system with emerge world.

In early 2017 I realised that my current job leaves me almost no spare time for a home-grown zoo of Linux distros. So I switched to Ubuntu (16.04 Xenial back then) and had been using Ubuntu until July 2023. In the lighthning talks session during the the EuroPython 2023 conferece I attended this year, someone mentioned Fedora as a nice alternative to Ubuntu for Python developers who need simplicity with the up-to-date versions of the upstream software. Given that I wasn’t happy with how Ubuntu upgrade to 22.04 went, the mess with snap packages, and ocassional performance issues I didn’t have time to troubleshoot properly, I decided to give Fedora 38 a try.

Now, 6 months and 1 system upgrade later, I can say that I am pretty happy with what Fedora Desktop provides. It’s simple, with the sound defaults like GNOME Desktop and Firefox browser. DNF package manager feels like at home without even looking up the man pages/help. The software is not the bleeding edge, but pretty up to date. Python support is excellent indeed: you get the latest interpeter version by default, but can install all the predecessors with ease, without even bothering with pyenv. Same with Java.

Fedora release cycle is 13 month. I have to admit, that my upgrade from Fedora 38 to 39 wasn’t fully seamless. I had to switch SELinux temporarily to the permissive mode to let the upgrade process update the boot loader image on my UEFI machine. But Fedora’s discussion forums are helpful and allowed me to fix the issue quickly.

All in all, Fedora feels like a right balance between the extremes: the very stable distributions with outdated software like Debian and the bleeding edge systems that are likely to break something during the upgrades like Arch Linux; between the blows-and-whistles-heavy popular distributions that seem to neglect the quality like late Ubuntu and the spartan, distros with survivalist-flavor like Gentoo.

So far so good! Let’s see how the upgrade to Fedora 40 will go.

GNU Stow repo with git submodules

2023-09-11T15:00:00+02:00

GNU Stow is a tool to manage symlinks from a single directory. That makes the tool a perfect candidte to keep configuration files (or dotfiles) in a VCS repository, such as a Git repo. This is where git submodules come in handy: oftentimes people maintain configurations as standalone repositories, e.g. configuration for Emacs. They can be plugged into a stow directory as a submodule.

Example stow directory

Let’s create a stow directory and initialise a git repository:

$ mkdir stow-example
$ cd stow-example/
$ git init
Initialized empty Git repository in /home/pilosus/git/stow-example/.git/

Now, let’s create a stow package for Emacs - a directory in a stow directory containing all the necessary files and directories to be installed into a target directory (which is $HOME or ~/). The package resides under emacs directory and will contain a .emacs.d directory as a git submodule:

$ mkdir emacs
$ cd emacs/
$ git submodule add git@github.com:pilosus/dot-emacs.git .emacs.d
Cloning into '/home/pilosus/git/stow-example/emacs/.emacs.d'...

Let’s add one more stow package as a git submodule, a ~/.clojure directory that contains tools and basic aliases for Clojure language:

$ cd ..
$ mkdir clojure
$ cd clojure/
$ git submodule add git@github.com:pilosus/dot-clojure.git .clojure
Cloning into '/home/pilosus/git/stow-example/clojure/.clojure'...

So now we have two stow packages, emacs and clojure installed as git submodules:

$ tree -R -a -L 2 -I '.git'
.
├── clojure
│   └── .clojure
├── emacs
│   └── .emacs.d
└── .gitmodules

Installing packages with Stow is as easy as this:

$ stow --target=$HOME emacs
$ stow --target=$HOME clojure

For bulk install */ can be used:

$ stow --target=$HOME --stow */

Stowing a package will create symlinks in the target directory ($HOME in this case) that reflect the hierarchy of files and directories in the package. E.g. for clojure package the hierarcy is the following:

$ cd clojure
$ tree -R -a
.
└── .clojure
    ├── deps.edn
    ├── .git
    ├── .gitignore
    ├── LICENSE
    ├── README.md
    └── tools
        ├── antq.edn
        ├── clj-watson.edn
        ├── new.edn
        └── tools.edn

It’s symlinked to a home directory that we used as a target directory when stowing:

$ ls -lth ~/.clojure
lrwxrwxrwx. 1 pilosus pilosus 29 Sep  5 15:51 /home/pilosus/.clojure -> git/stow-example/clojure/.clojure

Using git modules allows to support a standalone repository for configuration files so that they can be kept separately from the stow repo itself. This is a good solution given that a stow repo can contain sensitive configuration files such as inventory for Ansible or other authentication information.

Git submodules also allow to do all the normal git things such as checking out a certain commit or tag, pushing changes upstream, etc. That makes the package management easy and flexible.

My personal dotfiles

Here’s dotfiles, a repo with my personal config files organised with GNU/Stow.

Clojure: decision fatigue

2023-07-16T17:00:00+02:00

As someone who uses Clojure mostly for solo projects I often find myself in a situation when a decision on what libraries to use is hard. Clojure and its community seems to follow “There Is More Than One Way To Do It” path. Possibly often unintentiallally, because the core team is too small to ship some features before the community comes up with the missing parts. According to State of Clojure 2023 survey, Clojure is mostly used in smaller teams (of, presumably, experienced developers), which is probably the reason, why Clojure ecosystem also lacks opinionated tools that may save you some mental energy. All in all, these result in a decision fatigue when deciding on third-party libraries and tools.

Possible decisions

Enriched version of clojure-toolbox.com

When the quality of the popular search engines seems to be declining, a hand-picked collection may be a good solution. For Clojure, clojure-toolbox.com is a high quality categorised directory of libraries and tools for the language. And it seems to be well maintained. It has a few drawbacks though:

Projects are not rated
No hints on projects maintenance status
Projects description is hidden in tooltips

These are hard things. Not only it’s hard to compare dependencies that differ in their feature sets even under the same category. But also there could be no single source of statistics about number of downloads (Maven Central, Clojars, GitHub), the last release date, etc. Let alone the popularity isn’t necessary a sign of quality and the recent last release timestamp isn’t necessary a sign of good maintenance in a community with the strong commitment to backwards compatibility. Yet something simple would still be a huge relief to decision fatigue:

Number of stars on GitHub is a project uses it for hosting
The timestamp of the last commit on GitHub
Showing project’s description openly

It turned out, that 970 out of 1001 projects at the moment of writing had GitHub repo URL as a project link. That makes the approach mentioned above good enough!

Here’s how I enrich clojure-toolbox.com’s data with the GitHub stats:

Pull down clojure-toolbox.com DB (a YAML file stored in GitHub repo)
If a project has a GitHub link, get information about number of stargazers and the last commit push timestamp via GitHub API
Use clojure.core.async and GitHub access token to run operations in bulk
Format results in Markdown

Here’s an example code. And this is how the enriched catalogue looks like. Much more informative!

The same could possibly be done for The Babashka toolbox as it’s derived from the The Clojure Toolbox source code.

Clojurians Slack

Clojurians Slack is another great source of knowledge. Not only you can ask questions to some experienced Clojure programmers, but also search the Slack’s history.

Debugging and patching Clojure code running in production using socket server and REPL

2023-07-07T20:00:00+02:00

Common Lisp is well known for its interactivity. The story of NASA engineers who used remote REPL to patch Lisp code running 60 million miles away inside the interplanetary spacecraft is well known too. You can debug and patch Clojure code running in production using REPL too. This is where Clojure socket server comes in handy.

Practicle example

Let’s interact with the live web service written in Clojure and try to change the running program’s state using REPL and Clojure’s socket server. I assume you have Java 17 or later (earlier versions might work but not tested), Clojure 1.11 or later, git and curl installed on you computer to work through this tutorial.

Get sample code

Get dienstplan app (it’s opensourced):

$ git clone git@github.com:pilosus/dienstplan.git
$ cd dienstplan
$ git checkout ba1aa1f846bff8f690e14a6fbb8989ae73a10b53

Compile the code

Compile an uberjar. It produces app.jar in your working directory:

$ clojure -T:build uberjar :uber-file '"app.jar"'

Run the app with socket server

Let’s run the app with a Java option string -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}" needed to start a socket server with REPL on localhost with the port 5555 open. Environment variables are needed to start the app:

$ APP__DEBUG=false \
SLACK__TOKEN="xoxb-Your-Bot-User-OAuth-Token" \
SLACK__SIGN="Your-Signing-Secret" \
ALERTS__SENTRY_DSN="https://public:private@localhost/1" \
SERVER__PORT=8080 \
SERVER__LOGLEVEL=INFO \
DB__SERVER_NAME=localhost \
DB__PORT_NUMBER=15432 \
DB__DATABASE_NAME="dienstplan" \
DB__USERNAME="dienstplan" \
DB__PASSWORD="dienstplan" \
java -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}" -jar app.jar

Check if app is running properly

Let’s check if the app is working properly by making a HTTP GET request to its healthcheck API endpoint:

$ curl http://localhost:8080/api/healthcheck
{"status":"ok"}

$ curl -SsL -I http://localhost:8080/api/healthcheck | head -n 1
HTTP/1.1 200 OK

Connect to REPL

Now, let’s connect to the REPL socket server running alongside the app. Remember, it’s a pure REPL accessible via UDP connection, so your emacs CIDER’s nrepl, implementing more complex tooling protocol and working over TCP, won’t be able to connect to it! We can use simpler things like telnet or netcat (or nc) instead. For better line editing experience, like moving the cursor back and forth, using backspace, and history support, you may use readline library with rlwrap (readline wrap) CLI tool:

$ rlwrap nc localhost 5555
user=>

Access state of the running app

Now let’s tinker with our running app’s state from within the REPL! First, let’s take a look at the config initialized with the envorinment variables a few steps above. To do it we need to load app’s namespace dienstplan.config:

user=> (require '[dienstplan.config :as config])
nil

user=> config/config
{:application {:name "dienstplan", :version "latest", :env "production", :debug false}, :server {:port 8080, :loglevel "INFO", :access-log true, :block-thread true}, :slack {:token "xoxb-Your-Bot-User-OAuth-Token", :sign "Your-Signing-Secret"}, :alerts {:sentry "https://public:private@localhost/1"}, :db {:dbtype "postgres", :password "dienstplan", :minimumIdle 20, :username "dienstplan", :maxLifetime 1800000, :port 15432, :dbname "dienstplan", :connectionTimeout 10000, :host "localhost", :keepaliveTime 0, :maximumPoolSize 20}}

Patch the code of the runnning app

And now for something completely different! Let’s override some code in the running app! Let’s update the API handler /api/healthcheck that requested above with this:

(defmethod multi-handler :healthcheck
  [_]
  (log/warn
   "If I were to suggest that between the Earth and Mars there is a
   china teapot revolving about the sun in an elliptical orbit...")
  {:status 418
   :body {:status "I'm a teapot"}})

To do it, let’s use in-ns function to switch the current REPL’s namespace to dienstplan.endpoints where we want to override the method:

user=> (in-ns 'dienstplan.endpoints)
#object[clojure.lang.Namespace 0x50c83f27 "dienstplan.endpoints"]

dienstplan.endpoints=> (defmethod multi-handler :healthcheck
[_]
(log/warn
 "If I were to suggest that between the Earth and Mars there is a
china teapot revolving about the sun in an elliptical orbit...")
{:status 418
 :body {:status "I'm a teapot"}})
#object[clojure.lang.MultiFn 0x5d8ea5bb "clojure.lang.MultiFn@5d8ea5bb"]

Check if changes applied

Let’s make another HTTP GET request to the /api/healthcheck handler:

$ curl http://localhost:8080/api/healthcheck
{"status":"I'm a teapot"}

$ curl -SsL -I http://localhost:8080/api/healthcheck | head -n 1
HTTP/1.1 418 I'm a Teapot

Response body has changed, so did the HTTP status code. In the app logs we also see:

2023-07-07 18:52:00,665 [qtp1116648405-23] WARN  dienstplan.endpoints - If I were to suggest that between the Earth and Mars there is a
  china teapot revolving about the sun in an elliptical orbit...

Method has been sucessfully overriden!

Revert changes by restarting the process

Tinkering with the running code doesn’t change the code itself though. Restart the app process and fire another HTTP request to make sure all is back to normal:

$ curl http://localhost:8080/api/healthcheck
{"status":"ok"}

Conclusion

Socket server with REPL running along with the main app is a nice way to use various debugging tools and techniques the Clojure and third-parties provide. Code override won’t work if you use ahead-of-time compilation (doube-check your project settings, especially if you build uberjar with lein). But for on-the-fly compilation to JVM bytecode you can debug the running code without touching the code itself, right in the REPL, and revert the changes by simply restarting the app’s process.

Precautions

Running a socker server allows an intruder to connect to it over the network if an exposed port isn’t protected with a firewall, VPN and/or other access control tool. It may impose huge security risks on your app in production environment.

Patching code directly in the running app maybe the only available solution when your production environment is millions miles away and costs $150M. While patches themselves are easily reverted by restarting the process, your app’s state may be damaged irreversably.

Touching code in production is a great power, but also is a huge responsibility. Oftentimes risks of the live patching are higher than longer time needed to prepare and deploy the hotfix. You should know what you are doing.

kairos: crontab parsing library for Clojure

2023-07-07T12:00:00+02:00

I have been tinkering with Clojure since the late 2020. When I publish an open-source project in Clojure, it’s mostly about two goals: solving a real-world problem and learning some new concepts or tooling in Clojure ecosystem. kairos is a Clojure library for parsing crontab is no exception of this rule.

kairos parses vixie crontab format and produces a lazy sequence of java.time.ZonedDateTime objects for UTC timezone that are in the future.

Java interop

kairos doesn’t use any Clojure wrapper libraries to work with java.time classes, like cljc.java-time or clojure.java-time. Instead it uses Java interop directly, so that no dependencies are needed for the library.

The main lesson learned here: using type hints and enabling warnings on reflection can boost code performance to 6-8 times!

Plain (set! *warn-on-reflection* true) works well for reflection detection. But eastwood linter can do the job too, in addition to many other things.

Clojure CLI and deps.edn

kairos is my first project with deps.edn and Clojure CLI used from the day 1. In my previous projects I used lein. Leiningen is a fantastic integrated tool for all important project-related tasks from running tests to building an uberjar. I still think that lein is the best in class for Clojure beginners, who want to focus on Clojure code and not on the tooling. On the other hand, Clojure CLI allows you to be much more flexible when writing tasks for CI, e.g. create custom entrypoints in Clojure code for testing or deploying jars to Clojars. It also seems to be a first-class citizen when it comes to Clojure core team support.

Apart from official docs, Sean Corfield’s dot-clojure repo and Practicalli’s clojure-cli-config are both great sources of inspiration.

dienstplan: Slack bot for duty rotations

2022-01-16T00:59:00+01:00

dienstplan is a Slack bot for duty rotations. It’s dead simple: just a few commands to manage on-call duty rotations in your team’s Slack channels. The bot application follows the rule “Do One Thing and Do It Well”. It plays nicely with Slack reminders and workflows.

Usage example

Let’s create a rotation using dienstplan. Just pass in a create command followed by a rotation name, a list of the channel users in a rotation, and a rotation description:

@dienstplan create my-rota @user1 @user2 @user3
On-call engineer's duties:
- Process support team questions queue
- Resolve service alerts
- Check service health metrics
- Casual code refactoring-
 Follow the boy scout rule: always leave the campground cleaner than you found it

Once the rota is set up, the first user in the list becomes a current on-call person. Check it with a who command:

@dienstplan who my-rota

To change the current on-call person to the next one use rotate command:

@dienstplan rotate my-rota

The bot iterates over the users in the list order:

@user1 -> @user2 ->  @user3 -> @user1 ...

Now that you know the basics, let’s automate rotation and current duty notifications with Slack’s built-in /remind command. First, set up a reminder to rotate users weekly:

/remind #my-channel to "@dienstplan rotate my-rota" every Monday at 9AM UTC

Second, remind duties to a current on-call person:

/remind #my-channel to "@dienstplan who my-rota" every Monday, Tuesday, Wednesday, Thursday, Friday at 10AM UTC

Get help with:

@dienstplan help

Up & Running

Grab dienstplan from the GitHub repository along with user documentation, installation guide and deployment scripts.

pip-license-checker: detect third-party dependencies with “wrong” licenses

2021-09-07T09:10:00+02:00

Table of Contents

Intro

A software project may have hundreds of third-party dependencies. Even when available freely over the Internet, these dependencies may be distributed under licenses with some conditions, more or less restrictive. There are over a hundred popular free software and open-source licenses used nowadays. All of them may be roughly divided into a few license types depending on their conditions for usage, attribution, modification and distribution. Third-party dependencies under licenses of some types are good for your project, while others may result in license violations. License violations can pose high risks, both financial and reputation losses. Understanding what license types are good for your project’s 3rd-party dependencies is important. Once you know what are you looking for, check your dependencies regularly. pip-license-checker is a tool for automating license type checks that may help.

What is pip-license-checker?

pip-license-checker is a license compliance tool. It detects license types for your project’s third-party dependencies. The tool covers the majority of popular free/libre and open-source (FLOSS) licenses of all types: public domain, permissive, copyleft. It also detects proprietary licenses or EULA as well as technical problems obtaining the license for a dependency.

Originally developed with Python packages in mind, the tool operates in two modes:

Detecting license names and types for the list of Python dependencies
Detecting license types for any list of dependencies with their license names

The tool supports the output of some popular language-specific license detection plugins:

gradle-license-plugin for Andoid (Java),
cocoapods-acknowledgements of iOS (Swift, Objective-C),
license-checker for JavaScript,
lein-licenses for Clojure (Leiningen),

as well as universal CSV format.

The tool is distributed both as a:

command-line tool,
containerized application
GitHub Action,

so that it’s easy to use locally on your machine or incorporate into the project’s CI/CD pipeline.

Takeaways

pip-license-checker is a tool to check license types in your project’s third-party dependencies. It covers the majority of popular FLOSS licenses. It has integration with some popular license fetching plugins. You can easily incorporate the tool into CI/CD pipeline to automate license compliance checks.

Why license checks are needed?

Nobody develops software from scratch today. Software development has evolved from an artisan shop’s craft to a heavy industry assembly line. We, software developers, rely heavily on third-party code: libraries, packages, platforms, operating systems. Building something new has become, to a larger extent, assembling parts, applying some glue, implementing custom logic with the code written by someone else, someone outside of the project or the company. Most of such third-party dependencies today are freely available over the internet, both as executable code that a machine can run, and as human-readable code, we, developers, can use, copy and learn from.

But does being freely available over the network mean these third-party dependencies are really free? Free as in “free lunch”? Free as in “freedom”? Most of the dependencies are distributed under popular FLOSS licenses, e.g. Apache License, MIT License, Mozilla Public License or GNU General Public License to name a few. The licenses differ on the permissiveness/restrictiveness scale, ranging from the public-domain-like licenses with no conditions to the licenses requiring the authors of derivative works to give away the source code and use the same or compatible license under certain circumstances. Some dependencies are distributed as “freeware”, i.e. under proprietary end-user license agreements (EULA), usually with some permissions and conditions.

The conditions some licenses impose may be unacceptable for your personal project or your company. License violation is a legal risk. The court cases are real. Both financial and reputation losses may be high. In such cases preventing license violations is crucial to mitigate or avoid risks.

Sometimes choosing a “wrong” license for your FLOSS project may become an obstacle for its wide adoption (e.g. copyleft license for a project in a presence of many other alternative proprietary or free software projects of high quality). Checking licenses for project’s dependencies may help to understand your options when choosing the license most suitable for your goals.

Takeaways

Third-party dependencies freely available over the internet are usually distributed under some terms and conditions of FLOSS licenses or EULA. The conditions vary in their permissiveness. Some license types may be too restrictive to be used in your project.

Legal implications and reputation losses of license violations are real. To mitigate or avoid the risks regular license compliance checks are needed.

Why automating license compliance checks is needed?

The number of third-party dependencies (deps) in a project may be high: from tens in a tiny personal pet project to hundreds in a big one, to even thousands in the enterprise settings. As the project evolves the list of dependencies may change over time too.

The list of popular FLOSS licenses is also long. Over 100 licenses and counting! While some licenses are short and easy to understand, others are challenging reads that may require a lawyer qualification to grasp the conditions. Arguably, no average software developer ever read all of the FLOSS licenses that exist in the wild.

Increasing dependencies versions (so-called version bumps) in the project may also result in dependencies changing their licenses. This is the most relevant for the deps at the early stages of their development.

All in all, when managing a project’s 3rd-party dependencies, we need not only to comply with their licenses but do these license checks regularly, in an automated way. That requires a tool to be incorporated into the project’s continuous integration and continuous delivery (CD/CD) pipeline.

This is exactly what pip-license-checker is for.

Takeaways

Modern projects rely on 10s, 100s or even 1000s of third-party dependencies. Their list is changing over time as projects evolve. This is too much to check licenses manually.

Free Software Foundation and Open Source Initiative, the two most respected non-profit organisations for free/libre and open-source software, approved 100s of popular FLOSS licenses. Reading and understanding their conditions is a challenge for an average software developer. Expertise is needed.

Dependencies themselves may change their licenses over their lifetime too.

Automated regular license compliance checks are needed.

Who needs to check licenses?

Anyone who:

develops a proprietary project based on other FLOSS code
creates, contributes to, or maintains a FLOSS project

As an independent open-source developer, you may want to check your project deps’ licenses to make sure the project’s license itself complies with the libraries and packages it’s based on. That will help to avoid the risk of reputation losses or enable you to choose the right license for your project to make headway.

As a pro-profit company, you may want to make sure your proprietary software product doesn’t violate dependencies licenses to avoid legal risks, financial and reputation losses. As a startup, you may want to have license compliance in place before an investment round, valuation or acquisition.

Why pip-license-checker?

There are dozens of tools and plugins to detect license names for the third-party dependencies used in your project. But as it has been said above, for a large number of dependencies and an ever-growing number of open source licenses, having the list of license names may bring little or no value without legal expertise applied.

pip-license-checker brings some of the expertise needed by detecting license types, so that a user may decide what types are unacceptable in her/his project and get alerted whenever the licenses of “wrong” types are found among the project’s third-party dependencies.

What are license types?

Despite a large number of licenses used by third-party dependencies, all of them can be split into a relatively small number of categories.

Copyright or End-user license agreement

At the highest level, we may divide all the licenses into proprietary end-user license agreements (EULA) and free/libre and open-source (FLOSS) licenses. EULA dependencies may be freely available over the internet or even come with the source code available along with an executable object code, but their usage, distribution and modification may still be regulated in a more traditional copyright fashion: you need to ask authors for some permissions explicitly. By contrast, free software licenses grant the recipient of a piece of software rights to use, modify, redistribute the software.

FLOSS licenses, in turn, can be broken down into more types. They differ on the “permissiveness/restrictiveness” scale by the conditions they impose on someone who modifies and distributes the original software or its derivative works.

If you use a third-party dependency under a proprietary license agreement, it’s worth reading it as some unexpected conditions may apply even if the dependency is freely available over the network.

Public-domain-like licenses

The most permissive type of FLOSS licenses is public domain equivalent licenses. Zero-Clause BSD is a great example of public-domain-like licenses. This license type has no conditions, but may have a liability waiver.

Permissive licenses requiring attribution

Permissive licenses with attribution clause usually allow modification and redistribution under the license of choice with the only condition of preserving original copyright notice. Some licenses are indiscriminate about how exactly the attribution is to be done, while others require prominent notice, e.g. preserving the original notice in the source code and adding the copyright notice to the application’s legal section. Apache License and MIT License both fall under this license type.

Along with public-domain-like licenses, this license type is the most permissive of all FLOSS licenses, which makes them easy to be hijacked into a proprietary close-sourced code base or incorporated into a FLOSS project with no relicensing.

Copyleft

Copyleft is the next big category of licenses. The licenses of this type may also be called reciprocal as they require that modification and distribution of derivative works are done under the same or compatible license with the source code made available.

The copyleft licenses differ on the “permissiveness/restrictiveness” scale as well and can be broken down into the following fine-grained copyleft license types.

Weak copyleft

Weak copyleft licenses trigger the copyleft (reciprocal share-alike mechanism requiring to disclose the derivative work’s source code and to use the same or compatible license) when an original file or a library under the weak copyleft license is modified and redistributed. Simply using or incorporating the file or a library under weak copyleft into a larger work (e.g. by dynamic linking) and distributing it does not require giving away the source code or use the corresponding weak copyleft license. Mozilla Public License and Eclipse Public License are good examples of weak copyleft type of FLOSS licenses.

Third-party deps under weak copyleft licenses try to be business-friendly by allowing incorporating into a larger work but care about sharing the improvements made to the original work. Weak copyleft licenses may be unacceptable in some cases but very welcome in others.

Strong copyleft

The strong copyleft license type tries to go even further in its conditions that require to disclose the source code of the derivative works and require to use the same strong copyleft license. Combining your proprietary modules with a module under strong copyleft into a single executable file; dynamic linking of a library under strong copyleft license in your permissive FLOSS project; or even exchanging data between your program and a standalone program under strong copyleft in such a way that semantics of the communication are intimate enough, exchanging complex internal data structures, may be considered a work based on the software under the strong copyleft (derivative work) so that in case of distribution the source code must be given away and the same strong copyleft license must be used for the derivative work. GNU General Public License is the most illustrative example of a strong copyleft license type.

Third-party dependencies under strong copyleft licenses cannot be part of a proprietary code delivered to the outside users (unless a proprietor decided to give away a source code of the project and relicense it under a corresponding strong copyleft license). Private usage of a derivative work without distribution to outside users or customers (e.g. on your computer only, or on your company’s server only) is not considered a distribution and does not trigger the copyleft.

Copyleft over a network

The most restrictive license category of all copyleft licenses. Network copyleft license type picks up where the strong copyleft has left, i.e. it expands what distribution is and amends what private usage is by considering users interaction with a derivative work over a network as the work’s distribution. GNU Affero General Public License, a notable example of the copyleft-over-a-network license, aims at programs that are used on servers:

[…] if you run a modified program on a server and let other users communicate with it there, your server must also allow them to download the source code corresponding to the modified version running there.

Copyleft over a network license type may be too restrictive in many cases so that detecting it in your project’s dependencies is important.

License categorisation

Categorisation of popular FLOSS licenses into a few above-mentioned license types is the primary value of the pip-license-checker tool. Technically speaking, it’s just a mapping between regular expressions for license names to license types. But making up such a mapping requires reading and understanding the license texts.

Some licenses are derived from others, e.g. a license ZYX version 3.0 is a successor of the deprecated license ZYX version 2.0. In such cases, instead of reading both text categorisation could have relied on the latest version’s text along with the license’s official FAQ explaining the differences between the two versions.

Lacking lawyers support in the license categorisation process could have also resulted in some inaccuracies. Please, report issues regarding the license types.

Technical facts

pip-license-checker is a program written in Clojure. With quality in mind, the code has high test coverage and uses generative property-based testing. The tool’s own CI/CD pipeline includes a step for checking 3rd-party dependencies’ licenses with pip-license-checker.

Outro

Make use you know what license types are safe/desirable for your project’s 3rd-party dependencies. Is your project an open-source library for others to dynamically link in their software? Or a closed-source mobile app available over a popular digital distribution service as an executable? Or an embedded software for a device for wide public use? A website’s frontend software delivered to users when the website is requested via a browser? Or a backend software privately run in the background on your company’s server? In other words, what’s your project’s usage and distribution “profile” and what are the actions that are not acceptable for the project: source code disclosure, an obligation to change the license to reciprocal one, dependency’s authors’ attribution, etc.?

Once you know the answers, try to incorporate third-party deps license checks into your project’s CI/CD pipeline to detect licenses you are not safe to use. The pip-license-checker tool may help with that.

P.S. See the tool’s documentation in the pip-license-checker GitHub repository and the correspondent GitHub Action. Your feedback is welcome via email or GitHub Issues.

Disclaimer

pip-license-checker and this blog post are provided on an “as-is” basis and make no warranties regarding any information provided through them and disclaim liability for damages resulting from using them. Using pip-license-checker and this blog post does not constitute legal advice nor does it create an attorney-client relationship.

Addendum

Interview 2021-09-15

I’ve been interviewed by Yegor Bugayenko recently. We’ve talked about the pip-license-checker. As my very first public presentation of the project, the interview isn’t a perfect one. I’ve updated and extended this blog post in an attempt to fix all the flaws of my answers to the interviewer’s questions. Overall, the interview was good to show the project’s shortcomings and to give some directions for further development.

Book review: The Clean Coder by Robert C. Martin

2020-06-14T22:30:00+02:00

A Code Of Conduct For Professional Programmers, a subtitle of the book, says it all. Robert C. Martin answers the question: what does it mean to be a professional programmer.

If professionalism could be boiled down to a single aspect it’s responsibility. Responsibility for the following:

Product quality
Time estimates
Keeping competences and skills up to date
Understanding business goals and interests

Product quality

The book dedicates a few chapters to different aspects of keeping high standards of product development. The technical aspect of quality is about software testing:

hierarchy of testing (unit, component, integration, system, research testing)
test-driven development (TDD) as a way to deliver quality code, a safety net for refactoring, self-documentation for developers, a way to reduce code complexity and design program architecture more carefully.
acceptance testing as project documentation and criteria of what’s “done” (tests passing)
testing automation

The non-technical aspect of quality includes techniques to boost attention span. The author also dismantles some myths about productivity: coding late in the night, overtime work, etc.

Time estimates

Being responsible means you want to be predictable to your colleagues, deliver code on time. Correct time estimates are the single most important way to be predictable for a professional programmer.

The author just scratches the surface of the technical aspects of time estimation (PERT technique, planning poker, etc.). But he goes to great lengths to educate about the skill of saying “No” and “Yes”.

It’s okay for the business to try to get precise time estimates to eliminate the risks. It’s also okay for the business to wish to get things done quickly. Yet accepting unrealistic deadlines comes at a price for a programmer: work/life balance, productivity, and product quality, all of them deteriorate so that possible short-run benefits of working under pressure turn into a long-term disaster.

Keeping competences and skills up to date

A professional programmer must keep his/her technical and soft skills up to date. Technical skills require regular training. Just like professional musicians do their chores with scales, professional programmers should practice so-called code kata, programming or tech design problems, the author argues.

Learning new languages and systems in your spare time is also essential for a professional programmer.

Soft skills may include the ability to work in a team, do pair programming, or mentoring interns and colleagues.

Understanding business goals and interests

As a professional (i.e. responsible) programmer, you care about business interests. The business problems are your problems too.

Conclusion

The book has been published back in 2011. The industry has progressed quite a bit since then. Many bits of advice and techniques “Uncle Bob” suggests in his book now is a common practice. It’s unclear if the book will become a revelation for a reader in 2020. Yet it certainly encourages you to think about what does it mean exactly to be a professional and master your skills.

PostgreSQL: find sequences of same-parameter values within ordered groups

2020-04-18T18:00:00+02:00

Recently I’ve been solving a problem of finding database items that have to be grouped by one parameter (let’s say a name), then ordered by the timestamp within each group. The tricky part was to find groups where the last N most recent items all have the same second parameter (say, a flag). A colleague of mine pointed out to even more interesting variation of the same problem: find such groups that contain sequences of the size N of rows with the same flag.

Let’s create a table to illustrate the problem for the sequence size of 3 or more and a flag A:

CREATE TABLE names (
  name varchar(5),
  dt timestamp,
  flag varchar(1)
);

INSERT INTO names
  (name, dt, flag)
VALUES
  ('Name1', '2020-01-01 00:00:00', 'A'),
  ('Name1', '2020-02-01 00:00:00', 'A'),
  ('Name1', '2020-03-01 00:00:00', 'A'),
  ('Name1', '2020-04-01 00:00:00', 'B'),
  ('Name1', '2020-05-01 00:00:00', 'B'),
  ('Name1', '2020-06-01 00:00:00', 'A'),
  ('Name1', '2020-07-01 00:00:00', 'B'),
  ('Name1', '2020-08-01 00:00:00', 'A'),
  ('Name2', '2020-01-01 00:00:00', 'A'),
  ('Name2', '2020-02-01 00:00:00', 'A'),
  ('Name2', '2020-03-01 00:00:00', 'B'),
  ('Name2', '2020-04-01 00:00:00', 'A');

What we would like to have:

The result should be grouped by the name column
Within the group, the rows should be ordered by dt column
We are interested only in the groups containing sequences of 3 or more rows with the flag A

As we see in the data above Name1 has a sequence of three rows with the flag A (with dt 2020-01-01, 2020-02-01, and 2020-03-01). All other ordered by date consecutive rows with the same flag have two items or less. Thus the resulting SQL query should return Name1.

Whenever we need to group a set by the subsets of rows (windows) and then do something over these subsets, window function comes in handy. In this case, we would use ROW_NUMBER() that assigns a sequential integer to each row in a result set.

Postgres window functions have PARTITION BY clause that divides rows into multiple partitions. We can use more than one column here to group by the name and the flag columns. Let’s explore it:

SELECT
  *,
  ROW_NUMBER() OVER (PARTITION BY name ORDER BY dt ) AS name_part,
  ROW_NUMBER() OVER (PARTITION BY name, flag ORDER BY dt) AS name_flag_part,
  ROW_NUMBER() OVER (PARTITION BY name ORDER BY dt ) - ROW_NUMBER() OVER (PARTITION BY name, flag ORDER BY dt) AS seq
FROM names
ORDER BY name, dt

As you can see partitioning by the name column in ROW_NUMBER() function assigns consecutive row numbers within groups by the name column (a red bar for the Name1, a black one for Name2).

When partitioning by the two columns name and flag we have independent numbering within each name group for each flag subgroup:

green (flag A, name Name1)
blue (flag B, name Name1)
yellow (flag A, name Name2)
magenta (flag B, name Name2)

Given these numberings, how can we detect a sequence occurrence? We can subtract a row number for name and flag partitioning from the row number for the name column partitioning (see alias seq above). The subtaction gives each flag subgroup the same numbers that we can use for counting:

SELECT
  name,
  MIN(dt) AS min_dt,
  flag,
  COUNT(name) AS counter,
  seq
FROM (
  SELECT
    *,
    ROW_NUMBER() OVER (PARTITION BY name ORDER BY dt ) - ROW_NUMBER() OVER (PARTITION BY name, flag ORDER BY dt) AS seq
  FROM names
) numbered
GROUP BY name, flag, seq
HAVING COUNT(name) >= 3 AND flag = 'A'
ORDER BY name, min_dt

We do the following:

Find the difference between row numbers when partitioning by a name and partitioning by a name and a flag
Group the set by name, flag and the difference from the step above
Apply filtering for the grouped result to find sequences with flag A only with 3 or more rows
Order by the name and the timestamp of the first row in the sequence

Now that we found the solution let’s finalize the query to meet our goal:

SELECT
  DISTINCT name
FROM (
  SELECT
    name,
    flag,
    COUNT(name) AS counter,
    seq
  FROM (
    SELECT
      *,
      ROW_NUMBER() OVER (PARTITION BY name ORDER BY dt ) - ROW_NUMBER() OVER (PARTITION BY name, flag ORDER BY dt) AS seq
    FROM names
  ) numbered
  GROUP BY name, flag, seq
  HAVING COUNT(name) >= 3 AND flag = 'A'
) filtered
ORDER BY name

Summing up the post I would say that SQL and PostgreSQL specifically is a powerfull tool worth learning. Window functions allow doing some sophisticated analytics with a single SQL query.

workflow-tools: start using GitHub Actions at scale

2020-04-01T18:10:00+02:00

Microservice architecture may have dozens or even hundreds of lookalikes services that require similar CI/CD workflows. With infrastructure as code approach taken by the GitHub Actions, why not using code generation? Provisioning a repository for a new microservice may also be automated.

I’ve just opensourced an internal tool we use here at ANNA to automate GitHub Actions-based CI/CD workflow generation. workflow-tools is a collection of CLI tools used to:

generate GitHub Actions workflows from a Jijna2-template
set GitHub secrets (used by the workflows)

Examples

Let’s consider a real world example: setting up a GitHub Actions workflow for the workflow-tools repository itself. We need:

Generate a GitHub Action workflow using workflow_generator tool
Set GitHub Secrets the workflow needs using workflow_secret tool

Generating workflow

WORKFLOW_RUNNER_VERSION=ubuntu-latest \
workflow_generator \
docs/examples/master.tmpl \
~/PATH-TO-YOUR-REPO/.github/workflows/master.yml \
-e docs/examples/envfile

First, we define a Jinja2-template for the workflow (see a master.yml at line 3). Variables to be substituted should be marked up this way:

[[ workflow.your_variable ]]

When rendering the resulting file, workflow-generator tool substitutes the markup with the value of corresponding environment variable:

WORKFLOW_YOUR_VARIABLE

The environment variable can be set globally, for a single command run, or can be read from the envfile specified by the option flag -e (see an envfile at line 5).

Envfile comes in handy when a template uses many variables at once. It’s also easier to share variables between the templates using envfile. The envfile has the lowest precedence, it can be overridden (line 1).

Setting secrets

Now that we generated the workflow, let’s set a GitHub secret used by the template:

workflow_secret \
  --owner=anna-money \
  --repo=workflow-tools \
  --token="YOUR-PERSONAL-ACCESS-TOKEN" \
 update \
   --key=PYPI_PUSH_USER \
   --value=YOUR_SECRET_VALUE

First, we need to get a personal access token (see line 4). workflow_secret tool has multiple commands (see Tools autodocs). To set up new or update existing secret update command is used (line 5). The command accepts --key and --value options (lines 6, 7). workflow_secret also have tool-wide options used for each command: --owner (GitHub user, line 2), --repo (GitHub repository name, line 3) and --token.

Finally, let’s check what secrets are set for the repository:

workflow_secret \
  --owner=anna-money \
  --repo=workflow-tools \
  --token="YOUR-PERSONAL-ACCESS-TOKEN" \
 list

Why `workflow-tools`?

It’s a common practice to use templating in system administration, especially once DevOps practices have spread widely. Some tools Ansible template module have embedded templating mechanisms. It may be of great help where the tools originally fit. But it may be an overhead when applied to GitHub Actions configuration.

Other tools like envsubst have been there for a long time and proofed being useful. But one-size-fits-all approach may affect usability in a negative way.

The workflow-tools approach is to make Jinja2 standalone compiler with usability focused on the GitHub Actions workflows.

GitHub secrets setting tool from the workflow-tools package follows the same approach: simple CLI that does one thing and does it well, but not too general-purpose as it could be.

Who should use `workflow-tools`?

If you are using GitHub Actions and similar workflows are needed for multiple repositories, then workflow-tools package is for you.

Project links

Python logging: why printf-style string formatting may be better than f-strings

2020-01-24T09:10:00+01:00

Python provides more than one way to format strings: %-formatting, str.format(), string.Template and f-strings. What format developers use is the matter of personal aesthetic reasons rather than anything else. Still there are use cases where good old printf-style %-formatting may have advantages over other formats. Python’s logging module is one of these cases. Let’s see what are the main reasons to use %-formatting for logging in Python.

Python logging optimization: format string until it cannot be avoided

Python logging methods like debug get in message argument along with optional args and kwargs. Args used for message formatting. Formatting of these arguments is deferred until it cannot be avoided. That means the final message is not evaluated if its log level is below the logger’s log level. On the other hand, f-string is really an expression that evaluated at runtime and it lacks logging’s optimizations. Let’s see an example:

In [1]: import logging
   ...: logging.basicConfig(level=logging.INFO)
   ...: logger = logging.getLogger('TestLogger')

In [2]: class A:
   ...:     def __str__(self):
   ...:         print('Unnecessary slow computations are done here!')
   ...:         return self.__class__.__name__
   ...:
   ...: obj = A()

In [3]: logger.debug("log level below INFO with args: %s", obj)

In [4]: logger.debug(f"log level below INFO with f-string: {obj}")
Unnecessary slow computations are done here!

In [5]: logger.debug("log level below INFO with args: %s" % obj)
Unnecessary slow computations are done here!

In [6]: logger.info("log level INFO with args: %s", obj)
Unnecessary slow computations are done here!
INFO:TestLogger:log level INFO with args: A

As you can see in step 3 a message doesn’t get logged as the root logger’s level is higher than debug. Logging module’s inner optimization makes sure that the message is not formatted in this case. Compare this to step 6.

In step 4 the f-string is also not logged because of the low log level. But it’s formatted anyway. That’s why you see obj.__str__ called.

Please note, that logging optimization doesn’t work for "format string" % values format. You can see that in step 5. The magic applies for %-format message if and only if the values for message formatting are passed in as logger method’s arguments!

So step 3 shows the optimized way to do message formatting in logging.

By the way, logging module has other nice optimizations that worth learning.

Sentry integration with logging

Sentry is a popular error tracking solution. Sentry integrates with the Python logging module. Error aggregation is another great feature of Sentry. Sentry looks at the event’s stacktrace, exception, or message and group it with existing ones if they are the same. If you get a hundred of exceptions that are all the same they get grouped into one with nice statistics.

When integrating Sentry with Python logging, it’s important to use %-formatting, so that Sentry can group your messages. As an example, let’s log failed attempts to retrieve an URL:

logger.error("Failed to retrieve URL %s", url)

With printf-style formatting your messages will be grouped together no matter what value url variable holds. But in this case:

logger.error(f"Failed to retrieve URL {url}")

you get a separate event in Sentry for every single unique URL. If you get 1000 unique URL, your Sentry dashboard might get messy.

Conclusions

Although Python is not Perl, there’s still more than one way to do it. Which string formatting style to use is an open question. Your project may not use Sentry, or you may prefer f-strings for readability reasons. You may see deferred formatting in logging for printf-style strings as a futile micro-optimization that won’t improve your app performance. But still, there may be cases where using an oldie but goldie %-formatting is beneficial, even at price of string formatting inconsistency.

Updates

2020-01-24 21:35. Many thanks to symmitchry who stressed out the importance of passing values for a message to be formatted as arguments to logger’s method (e.g. args to logger.debug()). Probably, that wasn’t quite clear in the first edition of the post. I’ve made this point explicit.

2020-01-25 13:45. As a redditor pointed out in the post discussion a Python linter pylint has a rule W1202 that warns you when str.format is used in logging. It’s worth to mention a rule W1201 too, which is triggered for the cases like in step 5 (see the code snippet above).

Python third-party tools configuration: pyproject.toml vs setup.cfg

2019-12-26T12:10:00+01:00

A couple of years ago two PEPs have been proposed to fix some package building interface problems. Basically, in order to build a Python package we need to checkout source of the project, then install a build system and finally run it. PEP 517 defines the last step, including how to get the build system dynamically specify more dependencies that the build system requires to perform its job. The PEP 518 defines the second step, including a new configuration file pyproject.toml to be used by the building system.

As setuptools have been dominating in Python packaging field for a long time, many third-party tools adopted its configuration file setup.cfg too. Libraries for testing, static code analysis tools like linters or type checkers along with their custom configuration files often allow to use setup.cfg. After all, it’s INI-style configuration format, easy for humans to read and edit it, with no standard behind it though.

Moving tool settings to the setup.cfg as opposed to supporing a bunch of custom config files seems to be common practice. But now that we have the new PEPs, should we abandon setup.cfg in favour of the brand new pyproject.toml? Let’s see.

6 popular Python tools and their configs

Here are the 6 Python tools that I use daily both at my job and for the personal projects: pytest for testing, pytest-cov as a pytest plugin for coverage tool that calculates code coverage, mypy as a type checker, flake8 as a code style checker, black for code formatting, and isort for sorting imports.

In a table below you’ll find these tools along with their versions at the time of writing, popular and (arguably) universal config files as well as their native configuration file formats.

tool	version	`setup.cfg`	`pyproject.toml`	`tox.ini`	custom format
pytest	5.3.2	yes	no (WIP)	yes	`pytest.ini`
pytest-cov	2.8.1	yes	no	yes	`.coveragerc`, pytest conf
mypy	0.750.0	yes	no (won’t fix?)	no	`mypy.ini` (preferred)
flake8	3.7.9	yes	yes (unofficially)	yes	`.flake8`
black	19.10b0	no	yes	no	`--config` any TOML file
isort	4.3.21	yes	yes	yes	`.isort.cfg`

If there’s something to take in from the table, then it’s the following. First, there’s no consensus in the Python community over the usage of pyproject.toml. The most notable discussion involves Guido van Rossum in mypy issue on the GitHub. PEP 518 is all about configuration for package building systems like distutils, setuptools, flit, bento. It says nothing about configuration for tools outside of the package builing context. But what if package building process involves type checking or code formatting? Also, setup.cfg is a part of the setuptools which is a building system too. Still, people use it for third-party tools configuration outside of the package building context, e.g. for imports sorting checks in their CI pipelines. All in all, mypy has set a precedent for not supporting pyproject.toml on purpose, so that some other tools developers hesitate either to openly support the new config file like flake8 does, or just postpone implementation until a consensus is emerged in the Python community.

Secondly, there’s always a black sheep of the family. Being a great formatting tool that eventually has ended so many holy wars about code formatting, black doesn’t support anything except for TOML config files. Even if you are more inclined towards ol’ reliable setup.cfg, you still has a black sheep in your setup.cfg-compatible flock.

Conclusion

TOML is a great human-readable, machine-editable, strong typed, well-defined configuration format. Having Python- and setuptools-independent configuration file pyproject.toml is a great idea. Still, as the usage of pyproject.toml for non-package building context is not defined, we are in the xkcd how-standards-proliferate situation. Until consensus is finally set, it’s better off either use setup.cfg with an annoying exception for black, or stick to the tools custom config formats and files.

Python interview questions and answers

2019-12-15T14:00:00+01:00

Although the questions you find in this blog post are universal, they are crafted for a technical interview for a web developer position.

The questions have been tested in real life: I used them in more than 35 interview sessions. They allowed my team and me to hire three brilliant developers: a junior developer (ex-Yandex), a middle developer (ex-Yandex), and a senior one.

The questions have also been reviewed by my peers and got positive feedback. Some of my colleagues use these questions in their technical interviews. I also use the questions when consulting people about preparing for the interview or getting their first software job.

The questions are not unique. I’m not revealing any professional secrets by posting them. The single most valuable thing about these questions is the principle they are combined in an interview session. You can also use these questions to come up with your questions for the Python interview as an interviewer. Also, you can use them to prepare for an interview as an interviewee.

Table of Contents

Introduction
Python questions
Computer science questions
- Time complexity
- Balanced brackets in a Lisp-program problem
Databases
Testing
- Unit-testing overview
- How do you test a function that changes data in MySQL?
DevOps/SysAdmin skills
System Design and Architecture
- System design of pastebin-like service
Sources and useful links

Introduction

Premises

Software development is no longer an art, nor a lifelong quest, but rather an everyday office job, though highly-skilled one. That’s why a candidate’s abilities for teamwork, coping with chores are super important. But pure computer science, algorithms are in the background, whether you like it or not.
An algorithm-based technical interview is an analogue of IQ-tests that have been invented in the early XX century in the US for a quick military recruits categorization. IQ provides some insights about human intelligence but also has its flaws. Demonstrated algorithm skill also provides an insight about candidate’s intelligence but also has its flaws. Algorithm-based technical interviews are used by corporations to quickly process a huge queue of candidates, hire the ones they need and evaluate their grades and salary. Corporations everybody wants to work for can afford a false negative in hiring, i.e. not hiring the right person, as many candidates compete to be hired by these companies. If you are not one of these companies, then you compete with other companies for the candidates. So don’t pretend to be a Google if you are not Google.
Human intelligence neither is an “on/off”, nor a range. Rather, intelligence is a swiss army knife, a multitool. It may contain an excellent sharp blade for everyday chores, it may also have a screwdriver that merely works, and it may also have a corkscrew that you actually don’t need at all. Can we evaluate an overall score for such a multitool? Sure. But why would we? Unless you are Google, find a multitool with the blades and bits that solve your exact problems, not the abstract ones.
You need an interview to find out three things: a) why the candidate has left (or are ready to leave) his/her previous (current) job; b) what is he/she looking for at the new company; c) what value can he/she bring to your team?
Hiring error can be mitigated by the probationary period. Take it seriously. Set some goals (tasks) for probation. The same for the candidate: during probation decide if the company is the right choice. But hiring error is still an error that costs your company money and time. You should do your best to avoid this error. The same for the candidate. That’s why you never tell lies about your project and team in the interview. Still, there’s nothing wrong with presenting your project positively. Actually, it’s your duty if you want to hire someone in a highly competitive job market.
There are no two identical interviews even for the same grade. The technical interview is always improvisation. It’s an attempt to reveal a candidate’s boundaries of competence, strengths and weaknesses in a short period of time.

Candidates & CVs

Not everybody is a CV-ninja. Really
Too high salary expectations isn’t a reason to reject a candidate CV. You always can ask questions that will either lower the expectations or prove you are interviewing a genius.
(sarcasm mode on) Of course, an interviewer always knows best! (sarcasm mode off) But there are no solid grounds for reject candidate CVs with grammar mistakes or expressions of inexperience or quirks in their CVs. Asking inadequately complex questions is also not a good idea. The interview is not about you, it’s about the candidate.
Don’t afraid of people with some irrelevant background, e.g. designers, analysts, etc., who changed his/her profession to a technical one. Be honest, your boring REST API doesn’t require a university graduate program.
A candidate’s technology stack may not be 100% yours. Still, he/she may be a good match for your team. Should you hire someone with 3+ years of experience with Flask but no Django experience for your Django project? Probably you should.
The only way to hire someone quickly is to do as many interviews as you can.
Average interview time for an interviewer is about 2.5 hours (skim over the CVs, prepare for the interview, the interview itself, discussion with an HR specialist and a CTO)
Don’t make a technical interview for too long. 2 hour is enough. 1.5 hours is optimal.
Three interviews a week is a good pace. More than 3 is too much for the interviewer and his/her team. Less than 3 makes your search longer.
Average time for hiring a Python developer depends on the project, expected experience and your offer conditions. As a rule of thumb, 3 months from submitting an open position to HR department to the first working day is okay.
Not every software engineer is an introvert. There are people with outstanding “selling” skills who are prepared for typical interview questions, present themselves, but still lack solid knowledge of software development. Don’t let them trick you! Ask whiteboard programming questions!

Non-technical questions

What have you done in your previous/current project?
Describe your team line-up (frontend/backend developers, QA engineers, SRE/DevOps/SysAdmins, product/project managers, etc)?
How planning, code review, etc. are organized in your project? What you liked and disliked about it?
Why did you decice to leave (have already left) the project?
What do you expect from the new project (things to develop, processes, etc)?

Python questions

How language evolves

What’s new in the latest Python version? (Python 3.8 at the moment of writing)
How Python 3+ differs from Python 2.7? (What’s new in Python 3)
What new syntax has been introduced since Python 2.7? (minimal expectations: asyncio, type hinting)

Basic types and data structures in Python

What are the basic types in Python?
What are the basic data structures in Python?
All data types and structures in Python can be divided into two groups. Name these groups (mutable, immutable)

Problem. What lines of the following code raise exceptions? What expectations?

d = {}
d['a'] = 1
d[(1,2)] = 2
d[[1,2]] = 3       # TypeError: unhashable type: 'list'
d[(1,2,[3,4])] = 4 # TypeError: unhashable type: 'list'

Function default arguments

Problem. We have got a function:

def fun_default(val, lst: list=[]) -> list:
    lst.append(val)
    return lst

What the following print statements will print after the block of assignments is executed?

l1 = fun_default(1)
l2 = fun_default(2, [])
l3 = fun_default(3)
print(l1) # [1, 3]
print(l2) # [2]
print(l3) # [1, 3]

How would you fix the function so that print statements print [1], [2], [3] respectively?

def fun_default_fixed(val, lst: list=None) -> list:
    if lst is None:
        lst = []
    lst.append(val)
    return lst

What’s decorator?

What’s decorator?
What decorators from the standard library you know?
Tell about descriptor protocol (complex question)

Decorator that prints function execution time

We have got a function that queries a database. We want each function call to print time of its execution. Implement it with a decorator. Assume: decorated function may have any number of positional or keyword arguments.

import time

def measure(func):
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

Using decorator

How do you apply your decorator to a function fun using syntactic sugar and without it?

# with syntactic sugar
@measure
def fun(l: list) -> None:
    pass

# sugar free
fun = measure(fun)
type(fun) # function

Decorated function’s name

What is function docstring and name after decorator is applied?

fun.__name__  # 'wrapper'
fun.__doc__  # 'Wrapper function'

How do you return the original name and docstring to the decorated function?

from functools import wraps
import time

def measure(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

@measure
def fun(l: list) -> None:
    """My func"""
    pass

fun.__name__  # 'fun'
fun.__doc__   # 'My func'

How would you do that without functools.wraps?

import time

def measure(func):
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    wrapper.__doc__ = func.__doc__
    wrapper.__name__ = func.__name__
    return wrapper

Decorators for generators

What time do we get if we decorate a generator function like the following?

from typing import Generator, Any

def fun_gen(l: list) -> Generator[Any, None, None]:
    for i in l:
          yield i

Why time we see is so small? How would you measure the real time of iteration over the whole generator?

import time
from typing import Generator, Any

def measure_gen(func):
    def wrapper(*args, **kwargs):
        before = time.time()
        result = yield from func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

@measure_gen
def fun_gen(l: list) -> Generator[Any, None, None]:
    for i in l:
        yield i

Decorator with arguments

Rewrite the decorator from the previous question so that it could take in an argument for a limit. Execution time is printed if it exceeds the limit.

def measure(limit):
    # Don't forget to set limit when decorating a function!
    def outer_wrapper(func):
        def inner_wrapper(*args, **kwargs):
            before = time.time()
            result = func(*args, **kwargs)
            after = time.time()

            elapsed = after - before
            if elapsed > limit:
                print(f"Time elapsed: {after - before} seconds")

            return result
        return inner_wrapper
    return outer_wrapper

How do we decorate a function using syntactic sugar and without it?

new_measure = measure(3)
fun = new_measure(fun)
fun([1,2,3])

# sugar free
new_measure = measure(3)
fun = new_measure(fun)
fun([1,2,3])

Class variables

What’s a class variable?

Problem.

class Parent(object):
    x = 1

class Child1(Parent):
    pass

class Child2(Parent):
    pass

What will the following print statements print?

print(Parent.x, Child1.x, Child2.x)
# 1 1 1

Child1.x = 2
print(Parent.x, Child1.x, Child2.x)
# 1 2 1

Parent.x = 3
print(Parent.x, Child1.x, Child2.x)
# 3 2 3

List slices

Problem. What are the results of this code block execution?

l = [1, 2, 3, 4, 5]

l[1:2]  # [2]
l[:-1]  # [1, 2, 3, 4]
l[10:]  # []
l[10]   # IndexError

What’s list/dict comprehension?

Implement a list of even numbers in a range [0, 10) to the power 2 using list comprehension.

[i*i for i in range(10) if i % 2 == 0]

Implement the same using map and filter.

map(lambda x: x * x, filter(lambda x: x % 2 == 0, range(10)))

Generator expressions

What’s the easiest way to turn the above list comprehension into a generator (generator expression)?

(i*i for i in range(10) if i % 2 == 0)

What useful functions from functools do you know?

The question may be a follow-up for decorator questions and functools.wraps. functools is a great module with loads of useful things. What would be great to hear from the candidate is something like reduce and partial.

Asyncio, multithreading, multiprocessing

What’s the difference between a process and a thread?
Problem. We have got a list of 1000 URLs, we have to iterate over it and make GET requests. How would you do that? (IO-bound tasks -> asyncio, Tornado etc или multithreading)
Problem. We have got 1000 video files than should be rendered somehow using Python. How would you do that? (CPU-bound tasks -> multiprocessing)
Give an overview of asyncio or similar libraries for asynchronous programming. How does it work? (huge question, expectations are covered in Trio tutorial)

Python ecosystem

What do you use for background tasks in a Python project? (celery)
How do you pack your Python code? (wheels, eggs, what’s the difference?)
How do you document your Python project? (sphinx-doc)
What do you use for REST API documenting? (swagger)
What ORM do you use? (SQLAlchemy, Django ORM)
What static analysis tools and linters for Python code do you use? (mypy, flake8)
What web frameworks do you use? (Flask, Django, aiohttp, sanic, etc.)
What package management tools do you use? (pip, pipenv, poetry, etc.)

Computer science questions

Time complexity

What’s time complexity?
What’s Big O?
What’s time complexity for the following algorithm?

# O(n^2)
for x in n:
    for y in n:
         print(x, y)

What’s time complexity for each of these statements?

x in/not in s # where type(s) is list/tuple # O(n)
x in/not in s # where type(s) is set        # O(1)
list.append()                               # O(1)
list.insert()                               # O(n)
list[index]                                 # O(1)
list.sort                                   # O(n log n)

Check out other data structures time complexity if needed.

Balanced brackets in a Lisp-program problem

Problem. I like to code in Racket, which is a Lisp dialect that uses a lot of brackets. Implement a Python function that takes in text and returns True if brackets in the text are balanced, and returns False otherwise.

Examples of balanced brackets:

"()"
"(())"
"(custom-set-variables '(inhibit-startup-screen t) '(package-selected-packages (quote
(sql-indent magit dockerfile-mode yaml-mode gitlab markdown-mode lorem-ipsum jedi
elpy anaconda-mode afternoon-theme))))"
"(hello [world {!}])"

Examples of unbalanced brackets:

"("
")("
"())"
"({))"

def balance(s):
  """
  Return True if string's parentheses are balanced, False otherwise

  >>> balance("")
  True
  >>> balance("(())")
  True
  >>> balance("))((")
  False
  >>> balance("()()(())(()())")
  True
  >>> balance("(]")
  False
  """
  opening = {"(", "[", "{"}
  closing = {")", "]", "}"}
  mapping = dict(zip(opening, closing))
  stack = []
  for ch in s:
      if ch in opening:
          stack.append(mapping[ch])
      elif ch in closing:
          if (not stack) or (ch != stack.pop()):
              return False
  return not stack

If needed the function can be simplified to check only for round brackets.

Databases

Why RDBMS?

Problem. We are a startup with CTO, CEO and a couple of devs. We have no admins yet. We are developing authentication service (users registration, access rights, etc) in Python. Why do we need a database? Why not using a simple text file? Python has all we need to write to and read from a text file. We can just write user login and password hash to the file when the user is signing up. And then we read the credentials when the user is signing in in. It’s easy. It doesn’t require database admin work. Why not a text file?

If needed, the question can be put this way: what mechanisms of a relational database do you know, that are complex enough to implement, but are necessary for our case?

DB features that the candidate expected to mention:

index
relations
transactions

The candidate may be asked to elaborate on each of these features. For example, you can ask him/her about indices:

What’s time complexity for searching using the index?
Problem. In table candidates we have a is_interviewed column with possible values 0 or 1. How adding an index for this column will affect search performance?
What’s composite index (or multicolumn index)? What are the rules of using composite index?

MySQL/PostgreSQL queries

Problem. We have got service like iTunes where users can buy or rent a video. This is a PostgreSQL database schema for the service:

Write a SQL query to select top-20 movies starring actor John Doe. The query also selects:

Movie title
Movie language
Movie release year

Results should be ordered in descendant order by a release year.

(If help needed) use tables: film, language, film_actor, actor

SELECT f.title, l.name, f.release_year
FROM film AS f
JOIN language AS l ON l.language_id = f.language_id
JOIN film_actor AS fa ON fa.film_id = f.film_id
JOIN actor AS a ON a.actor_id = fa.actor_id
WHERE a.first_name = 'John' AND a.last_name = 'Doe'
ORDER BY f.release_year DESC
LIMIT 20;

Write a SQL query to select top-10 customers who paid most, but not less than 100. The query also selects:

Customer’s ID and full name
Customer sum of payments

(If help needed) use tables: payment, customer

SELECT c.customer_id, c.first_name, c.last_name, SUM(p.amount) AS money
FROM payment AS p
JOIN customer AS c ON c.customer_id = p.customer_id
GROUP BY c.customer_id
HAVING SUM(p.amount) >= 100
ORDER BY money DESC
LIMIT 10;

NoSQL Overview

What’s NoSQL?
What types of NoSQL do you know?
What NoSQL DB did you use?

Minimal expected answer: NoSQL is a database for collections of data. Data is denormalized, JOIN is not supported (usually implemented in your app).

Types of NoSQL DBs:

key-value store (hash table implementation: Redis, AeroSpike)
document-store (like key-value but with documents, e.g JSON, for values: MongoDB)
wide column store (HBase, Cassandra)
graph database (Neo4j)

Redis Overview

What’s Redis is used for?
Why it’s so popular for caching layer?
How Redis is different from MySQL when it comes to data storage?
What happens if a machine that runs Redis server is rebooted?
What high availability mechanisms for Redis do you know?

Expectations: Redis is an in-memory key-value DB. Data snapshot mechanism can be used in Redis to dump data to the disk. Redis Sentinel is a mechanism for high availability.

Testing

Unit-testing overview

What’s unit-testing?
How it’s different from integration or functional testing?
What libraries did you use for testing in Python? (unittest, pytest) How they are different from each other?
When you write a unit-test? (in the existing project, in the new project, when fixing a bug, when implementing a new feature)

How do you test a function that changes data in MySQL?

Problem. We’ve got a function that executes a INSERT SQL query in the test database. Other developers in your team use the same database. We need to write a unittest for the function. But we don’t want to change test DB on each test run. How do you do it?

mock
transaction rollback

DevOps/SysAdmin skills

Broken response from google.com

Problem. When I open a browser, go to https://www.google.com and see the raw response, it’s normal text.

Here I do a GET request using curl, but a response seems to be broken.

> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.58.0
> Accept: */*
> Accept-Encoding: gzip, deflate

%�j��~d+.K�W��me`�,x�x�m�G�67��`$#������~{d���]�.'����
����Q���Mʥu���<�O),�&�����,���E�.���_�
&�I\��p��#]�,�����K+����p�vk
j���k"�fe�=�`�J4�I/��m6{�E���;3+�v�BT1T��(W��J��m�yO��
�j��

Why is it so?

curl -v "https://www.google.com" --header "Accept-Encoding: gzip, deflate" --output -

IP addresses top from Nginx log

Problem. Our service is under high load because of DDoS-attack. It makes hard for our legitimate customers to get a response from the service. We want to ban IP addresses bots get used of. To do so we want to make a top of IP addresses with the most requests. We ask our admins to provide web server logs as a file nginx.log.

Using console GNU utils create a file with top IP-addresses with most requests in descending order. Assume that IP-address is in 9th column in the log, columns are separated by t.

awk '{print $9}' nginx.log | sort | uniq -c | awk '{print $1","$2}' | sort -r -t, -nk1 > bots.csv

Docker

What’s a docker container?
What’s the difference between docker and virtualbox or other virtual machines?
Why docker? We can just copy files to the real hardware machine
Why not using virtualenv instead of Docker?

System Design and Architecture

Parental advisory: highly opinionated content (35+)

System design of pastebin-like service

User story (business description)

A user submits text (copypasta)
Text is saved, the user gets a unique URL (base serive URL + short path)
User goes to the URL and sees his/her original text
Text is stored until it’s expired
Service deletes expired texts
Once the text is expired, it’s not accessible anymore by its URL
Service is operating under high load
A product owner needs monthly statistics for URLs visits

Assume

Copypasta can be in text format only
The product owner doesn’t need realtime statistics
Service gets 10 million new copypastas a month (3.85 RPS)
Service has 100 million copypasta reads a month (38.5 RPS)
We need capacity planning (hardware planning) for 3 years

Hints

Average text size is 1Kb
Short URL path takes 7 bytes in DB
URL expiry (or TTL) takes 4 bytes in DB
URL timestamp takes 5 bytes in DB
File path to a copypasta takes 255 bytes in DB
In total: 1.27 KB per copypasta or 1.27 Gb of copypasta a month

Capacity planning for 3 years

36 * 12.7 = 457 Gb of copypasta
36 * 10 = 360 million URLs

Service architecture

  Client
    |
    V
Load Balancer
    |
    V
 Web Server (Reverse Proxy, i.e. Nginx)
  |      |
  V      V
Write   Read                                Analytics (MapReduce)
API     API---------------------------       |         |
 |       |        |                  |       |         |
 |       V        |                  |       |         |
 |      Cache     |                  |       |         V
 ------------------------------      |       |       SQL for Analytics
 V                |            |     |       |
SQL-----------    |            V     V       V
 |_Master    |    |           Object (File) Store
 |_Slave     V    |
            SQL   V
 ^              Read-only Replica
 |
 |
Periodic Task
(remove expired)

Write API

Generate a URL, check for unique constraint in SQL
If the URL is unique, make an INSERT into DB table, otherwise go to 1.

shortlink char(7) NOT NULL
expiration_in_minutes int NOT NULL
created_at datetime NOT NULL
paste_path varchar(255) NOT NULL
PRIMARY KEY(shortlink)
CREATE INDEX shortlink_idx ON pastes (shortlink) USING BTREE;;
CREATE INDEX created_at_idx ON pastes (created_at) USING BTREE;;

Unique URL path generation

Generate a random string md5(“user IP address string” + “timestamp string”)
Encode string from 1 with Base 62 (like Base 64 but without escape characters)

def base_encode(num, base=62):
    digits = []
    while num > 0
      remainder = modulo(num, base)
      digits.push(remainder)
      num = divide(num, base)
    digits = digits.reverse

Get first 7 characters from base62 string: it’s 62**7 possible short paths (which is much more than we need in our 3 years plan, i.e. 360 million URLs)

url = base_encode(md5(ip_address+timestamp))[:URL_LENGTH]

Request Write API

$ curl -X POST https://pastebin.com/api/v1.0/paste \
       --data '{ "expiration_in_minutes": "60", "paste_contents": "My paste" }'

Write API Response

{
    "shortlink": "xJlsmeT"
}

Request Read API

$ curl https://pastebin.com/api/v1.0/paste?shortlink=xJlsmeT

Read API Reponse

{
    "paste": "My paste"
    "created_at": "YYYY-MM-DD HH:MM:SS"
    "expiration_in_minutes": "60"
}

Statistics

Since we don’t need realtime analytics, let’s use Nginx logs and process them in MapReduce

class HitCounts(MRJob):

    def extract_url(self, line):
        """Extract the generated url from the log line."""
        ...

    def extract_year_month(self, line):
        """Return the year and month portions of the timestamp."""
        ...

    def mapper(self, _, line):
        """Parse each log line, extract and transform relevant lines.

        Emit key value pairs of the form:

        (2016-01, url0), 1
        (2016-01, url0), 1
        (2016-01, url1), 1
        """
        url = self.extract_url(line)
        period = self.extract_year_month(line)
        yield (period, url), 1

    def reducer(self, key, values):
        """Sum values for each key.

        (2016-01, url0), 2
        (2016-01, url1), 1
        """
        yield key, sum(values)

Sources and useful links

Toptal Python Interview Questions
Python Interview Questions (junior, middle, senior)
System design interview
The Python Language Reference
PostgreSQL Tutorial

Вопросы и ответы для собеседования веб-разработчика на Python

2019-12-08T14:57:00+01:00

Ниже вы найдете принципы проведения и вопросы для технического собеседования на позицию Python-разработчика. Хотя вопросы довольно универсальны, больше всего они подойдут для области веб-разработки.

Вопросы прошли испытания боем: с их помощью было прособеседовано больше 35 кандидатов. Из них 3 (2 экс-Яндекс) стали частью моей команды в 2018 году, и еще 1 кандидат вышел в параллельную команду.

Вопросы получили положительный отклик коллег: люди используют их в каком-то объеме для собеседований в своих компаниях. Я также использую эти вопросы для консультации людей, которые готовятся к прохождению технического собеседования.

В то же время вопросы едва ли можно считать уникальными. Здесь нет никаких профессиональных тайн. На мой взгляд, основное преимущество вопросов — их система и принципы. На их основе можно составлять свои интересные и полезные вопросы для проведения собеседования, скроенные для ваших нужд. Или готовиться к техническому интервью в качестве кандидата.

Содержание

Введение
Python
Алгоритмы
- Временная сложность
- Проверка сбалансированности скобок в Lisp-программе
Базы данных
Тестирование
- Общие представления о unit-тестировании
- Как протестировать функцию, которая изменяет запись в MySQL?
DevOps/Администрирование
Архитектура
- Проектирование сервиса копипасты pastebin
Полезные ссылки
Errata

Введение

Предпосылки

Разработка софта больше не искусство, не дело жизни, а обычная, хотя и высококвалифицированная офисная работа. Поэтому особенно ценно в кандидате то, как он работает в команде, как справляется с повседневной работой. Как бы банально и избито это ни звучало. А вот гениальные технические озарения и хитрые алгоритмы, на втором плане. Те, для кого на первом, сами знают, каким должно быть техническое собеседование.
Алгоритмическое собеседование это аналог IQ-тестов, изобретенных американскими учеными в начале XX века для быстрой категоризации больших масс призывников по родам войск. IQ дает представление об интеллекте человека, но не исключает ошибок. Умение решать алгоритмические задачи тоже дает представление об интеллекте человека, но не лишено недостатков. Алгоритмические собеседования нужны крупным компаниям, которым требуется быстро отобрать нужных людей из огромных масс кандидатов и установить на входе грейд и зарплату. Они могут позволить себе ошибку ненайма, потому что кандидаты конкурируют за позиции в этих компаниях, а не наоборот, как большинстве других случаев. Не стройте из себя Google, если вы им не являетесь.
Интеллект человека это не бинарное “есть/нет” и не диапазон. Интеллект — это что-то вроде швейцарского армейского ножа-мультитула. В нем может быть острое и практичное лезвие для повседневного использования, не слишком удобный штопор, которым все же можно пользоваться, и бесполезная отвертка, которой не сделаешь ничего. Можно ли дать интегральную оценку такому мультитулу? Наверное, можно. Но непонятно, зачем. Если вы не Google, конечно. Подбирайте нож под свои задачи и исходя из своих возможностей.
Собеседование нужно, чтобы понять, почему человек ушел (или готов уйти) с текущего места работы, что он ищет на новом месте и что он может предложить команде.
Ошибка найма обычно компенсируется испытательным сроком. Нужно отнестись к нему серьезно и выдать задание на этот период. Аналогично со стороны сотрудника. И все-таки ошибка найма — это ошибка, которая стоит вам времени и денег. Лучшее ее не совершать. Аналогично для сотрудника. Поэтому врать про проект не стоит. Но уметь его красиво преподнести нисколько не зазорно. Напротив, в условиях конкуренции компаний за кандидатов, это ваша прямая обязанность.
Не бывает двух одинаковых собеседований даже для кандидатов одного уровня. Техническое собеседование это всегда импровизация, попытка найти границы компетентности и сильные стороны кандидата за короткий промежуток времени.

Отбор кандидатов

Не все люди умеют составлять резюме.
Слишком высокие зарплатные ожидания у интересного кандидата – не повод отказываться от собеседования. Всегда можно позадавать вопросы, которые снизят неоправданные зарплатные ожидания. Или подтвердят, что перед вами гений.
Интевьюер, несомненно, самый умный и замечательный. Но это не повод забраковывать резюме с грамматическими ошибками (как учат вас горе HRы), смешными формулировками с проявлениями юношеской неопытности и всяким таким прочим. Задавать неоправданно сложные вопросы тоже не стоит. Оставьте высокомерие при себе.
Не нужно бояться звать на собеседование человека, который раньше работал дизайнером/юристом/прорабом на заводе, но переучился на разработчика. Будем же честны, разрабатывать вашу скучную REST API много ума не надо, а профильное образование нужно далеко не везде.
У кандидата может не быть 100% попадания в ваш технологический стэк, что не помешает ему в нем очень быстро разобраться. Скорее всего, неразумно отвергать кандидата с опытом работы с Flask 3 года, но без опыта работы с Django, на ваш заурядный Django-проект. Ну правда.
Единственный способ быстро найти разработчика: провести как можно больше собеседований
Среднее время, которое затратит интервьюер на 1 собеседование: ~2.5 часа (отсмотр резюме, подготовка к интервью, само интервью, общение с HR и CTO).
Не стоит делать техническое интервью дольше 2 часов. Идеально 1.5 часа.
3 собеседования в неделю – это хороший темп. Больше – тяжело (физически для интервьюера, для HR в смыле поиска кандидата и для проекта в смысле отсутствия лида во время собеседований). Меньше – подбор сотрудника затянется.
Среднее время поиска Python-разработчика зависит от проекта, грейда и того, что может предложить ваша компания. В среднем 3 месяца от размещения заявки в HR до первого дня работы нового сотрудника это нормально.
Ваш главный союзник в общегуманитарных вопросах собеседования - ваш HR.
Не все разработчики – это тихие интроверты. Есть достаточное количество людей, которые умеют говорить, продавать себя и выучивать ответы для собеседований, не имея глубоких знаний. Не дайте себя обмануть, обязательно давайте задачки.

Гуманитарные вопросы

Что кандидат делал на прошлых местах работы. Какую позицию занимал?
Какой состав команды был на прошлых проектах (фронт, бэк, QA, админы, менеджеры)?
Как был организован процесс планирования, постановки задач и ревью? Какие в этом были плюсы и минусы?
Почему кандидат решил уйти (или уже ушел) с прошлого места работы?
Что хотелось бы от нового проекта? В смысле задач и процессов?

Python

Проверка знаний, как развивается язык

Расскажи, что появилось в последней версии Python? (Python 3.8 на момент создания статьи)
В чем состоят основные различия между Python 2.7 и 3.0
Какие синтаксические изменения произошли в языке после Python 2.7? (хочется услышать как минимум об asyncio и type hinting)

Основные типы и структуры данных в Python

Назови основные типы данных в Python
Расскажи, какие структуры данных есть в Python
На какие два больших класса можно разделить типы/структуры данных? (mutable, immutable)
Зачем вообще нужны immutable объекты? Почему нельзя везде использовать mutable?
Какие строки в примере ниже выбросят исключение? Какие исключения?

d = {}
d['a'] = 1
d[(1,2)] = 2
d[[1,2]] = 3 # TypeError: unhashable type: 'list'
d[(1,2,[3,4])] = 4 # TypeError: unhashable type: 'list'

Порядок вычисления дефолтных аргументов функций

У нас есть функция:

def fun_default(val, lst: list=[]) -> list:
    lst.append(val)
    return lst

Какие значения будут иметь следующие списки после выполнения всего блока присваиваний:

l1 = fun_default(1)
l2 = fun_default(2, [])
l3 = fun_default(3)
print(l1) # [1, 3]
print(l2) # [2]
print(l3) # [1, 3]

Как поправить эту функцию, чтобы ответ был [1], [2], [3] соответственно?

def fun_default_fixed(val, lst: list=None) -> list:
    if lst is None:
        lst = []
    lst.append(val)
    return lst

Что такое декоратор?

Что такое декоратор?
Какие декораторы из коробки знаешь?
Расскажи про протокол descriptor (сложный вопрос, можно раскручивать долго)

Декоратор, выводящий в stdout время выполнения функции

У нас есть функция, которая обращается в БД. Хочется, чтобы каждый вызов этой функции показывал в консоли время ее выполнения. Реализуй этот функционал с помощью декоратора. Допущение: декорируемая функция может иметь любое количество позиционных и непозиционных аргументов.

import time

def measure(func):
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

Применение декоратора

Как можно применить декоратор к функции fun, используя синтаксический сахар и без него?

# with syntactic sugar
@measure
def fun(l: list) -> None:
    pass

# sugar free
fun = measure(fun)
type(fun) # function

Имя декорированной функции

Какое имя и docstring будут у функции после применения к ней декоратора?

fun.__name__  # 'wrapper'
fun.__doc__  # 'Wrapper function'

Как вернуть задекорированной функции оригинальное имя и doctring?

from functools import wraps
import time

def measure(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

@measure
def fun(l: list) -> None:
    """My func"""
    pass

fun.__name__  # 'fun'
fun.__doc__  # 'My func'

Как добиться того же эффекта без functools.wraps?

import time

def measure(func):
    def wrapper(*args, **kwargs):
        """Wrapper function"""
        before = time.time()
        result = func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    wrapper.__doc__ = func.__doc__
    wrapper.__name__ = func.__name__
    return wrapper

Декоратор для генераторов

Какое время будет выводиться, если применить декоратор к генератору?

Допустим, мы применим написанный выше декоратор к такой функции:

from typing import Generator, Any

def fun_gen(l: list) -> Generator[Any, None, None]:
    for i in l:
          yield i

Почему в консоли время, выведенное декоратором, такое маленькое? Как замерить время итерации по всему генератору?

Этот вариант декоратора прокрутит генератор до конца и выведет общее время итерирования:

import time
from typing import Generator, Any

def measure_gen(func):
    def wrapper(*args, **kwargs):
        before = time.time()
        result = yield from func(*args, **kwargs)
        after = time.time()
        print(f"Time elapsed: {after - before} seconds")
        return result
    return wrapper

@measure_gen
def fun_gen(l: list) -> Generator[Any, None, None]:
    for i in l:
        yield i

Декоратор с аргументами

Перепиши декоратор из задачи выше так, чтобы в него можно было передать аргумент, который будет устанавливать пороговое значение, начиная с которого будет выводиться время исполнения декорируемой функции:

def measure(limit):
    # Don't forget to set limit when decorating a function!
    def outer_wrapper(func):
        def inner_wrapper(*args, **kwargs):
            before = time.time()
            result = func(*args, **kwargs)
            after = time.time()

            elapsed = after - before
            if elapsed > limit:
                print(f"Time elapsed: {after - before} seconds")

            return result
        return inner_wrapper
    return outer_wrapper

Покажи, как использовать этот декоратор с синтаксическим сахаром и без него.

# syntactic sugar
@measure(3)
def fun(l: list) -> None:
    pass

# sugar free
new_measure = measure(3)
fun = new_measure(fun)
fun([1,2,3])

Переменные класса

Что такое переменная класса?

Даны следующие классы:

class Parent(object):
    x = 1

class Child1(Parent):
    pass

class Child2(Parent):
    pass

Что выведут следующие строки?

print(Parent.x, Child1.x, Child2.x)
# 1 1 1

Child1.x = 2
print(Parent.x, Child1.x, Child2.x)
# 1 2 1

Parent.x = 3
print(Parent.x, Child1.x, Child2.x)
# 3 2 3

Слайсы в списках

Как исполнятся следующие строки?

l = [1, 2, 3, 4, 5]

l[1:2]  # [2]
l[:-1]  # [1, 2, 3, 4]
l[10:]  # []
l[10]  # IndexError

Что такое list/dict comprehensions?

Создай список квадрат четных чисел от [0, 10), используя list comprehension

[i*i for i in range(10) if i % 2 == 0]

То же самое, используя map и filter?

map(lambda x: x * x, filter(lambda x: x % 2 == 0, range(10)))

Generator expressions

Как превратить первый пример в generator (generator expression)?

(i*i for i in range(10) if i % 2 == 0)

Какие полезные функции из модуля functools ты знаешь?

Этот вопрос можно задавать после вопроса про декораторы и functools.wraps. В functools много всякого интересного. Хотелось бы услышать про reduce и особенно про partial.

Asyncio, multithreading, multiprocessing

В чем разница между процессом и трэдом?
Есть лист из 1к урлов, по каждому из них нужно сделать GET запрос. Как это сделать быстрее и дешевле? (IO-bound tasks -> asyncio, Tornado etc или multithreading)
Есть 1к видеофайлов, которые надо отрендерить, или 1к программ, которые надо скомпилировать. Как это сделать быстрее? (CPU-bound tasks -> multiprocessing)
Как устроен asyncio или аналогичные библиотеки для асинхронного программирования в Python? (огромный вопрос, который можно превратить в целое собеседование. Хочется как минимум понимания на уровне туториала Trio)

Экосистема языка

Какие решения существуют для выполнения фоновых отложенных задач? (celery)
Как сделать package из Python-кода? (wheels, eggs, в чем отличия?)
Какие есть решения для документирования кода на Python? (sphinx-doc)
Какие есть решения для документирования REST API (swagger)
Назови ORM, с которыми ты работал (SQLAlchemy, Django ORM)
Какие статические анализаторы кода, линтеры для Python ты знаешь? (mypy, flake8)
Какие web-фреймворки тебе известны? (Flask, Django, aiohttp, sanic, etc.)
Какие package management tools ты знаешь? (pip, pipenv, poetry, etc.)

Алгоритмы

Временная сложность

Что такое временная сложность (time complexity)?
Что такое большая O?
Какая временная сложность вот такой операции:

# O(n^2)
for x in n:
    for y in n:
         print(x, y)

Какова временная сложность для следующих стейтментов?

x in/not in s # где type(s) is list/tuple # O(n)
x in/not in s # где type(s) is set        # O(1)
list.append()                             # O(1)
list.insert()                             # O(n)
list[index]                               # O(1)
list.sort                                 # O(n log n)

Можно еще поспрашивать временную сложность операций над другими структурами данных.

Проверка сбалансированности скобок в Lisp-программе

Я обожаю программировать на Racket – это диалект Lisp, который славится большим количеством скобочек. Напиши функцию на Python, которая будет получать на вход текст (строку) и возвращать True, если скобки в тексте сбалансированы и False в противном случае.

Примеры сбалансированных скобочек:

"()"
"(())"
"(custom-set-variables '(inhibit-startup-screen t) '(package-selected-packages (quote
(sql-indent magit dockerfile-mode yaml-mode gitlab markdown-mode lorem-ipsum jedi
elpy anaconda-mode afternoon-theme))))"
"(hello [world {!}])"

Примеры несбалансированных скобочек:

"("
")("
"())"
"({))"

def balance(s):
  """
  Return True if string's parentheses are balanced, False otherwise

  >>> balance("")
  True
  >>> balance("(())")
  True
  >>> balance("))((")
  False
  >>> balance("()()(())(()())")
  True
  >>> balance("(]")
  False
  """
  opening = ("(", "[", "{")
  closing = (")", "]", "}")
  mapping = dict(zip(opening, closing))
  stack = []
  for ch in s:
      if ch in opening:
          stack.append(mapping[ch])
      elif ch in closing:
          if (not stack) or (ch != stack.pop()):
              return False
  return not stack

Если кандидату трудно придумать решение для скобочек разных типов, можно упростить задание до поверки, например, только круглых скобочек.

Базы данных

Зачем нужна RDBMS?

У нас стартап из CTO, CEO и одного разработчика. Админов нету. Мы пишем сервис аутентификации – регистрация пользотелей, проверка прав – на Python. Зачем нам нужна БД? Почему бы не использовать текстовый файл? Стандартная библиотека языка имеет все необходимое для работы с файлами. Записываем логин и хэш пароля при регистрации, читаем при запросе доступа. Это просто. Не требует админитрирования, избавляет от зависимостей. При сложностях, можно переформулировать вопрос так – какие существуют механизмы реляционной БД, достаточно сложные для их самостоятельно реализации, но необходимые для нашего случая?

От кандидата хочется услышать упоминания следующих механизмов:

индекс
связи таблиц
транзакции

Про каждый механизм можно поспрашивать отдельно. Например, про индекс можно спросить:

временная сложность для поиска по индексу
в таблице candidates есть колонка is_interviewed с возможными значениями 0 и 1. какой прирост произодительности для поиска по этой колонке даст индекс для нее?
что такое составные индексы? есть ли какие-то правила для запросов по колонкам, участвующим в составном индексе?

Запросы в MySQL/PostgreSQL

У нас есть сервис по типу iTunes, где можно арендовать/покупать видео. Вот схема таблиц реляционной базы, положим, в PostgreSQL

Напиши запрос, который выведет топ-20 фильмов, в которых играет актер John Doe. Запрос должен выводить:

Название фильма
Язык издания
Год выпуска

Результаты нужно упорядочить по убыванию года выпуска.

(Если нужна подсказка) на основе таблиц: film, language, film_actor, actor

SELECT f.title, l.name, f.release_year
FROM film AS f
JOIN language AS l ON l.language_id = f.language_id
JOIN film_actor AS fa ON fa.film_id = f.film_id
JOIN actor AS a ON a.actor_id = fa.actor_id
WHERE a.first_name = 'John' AND a.last_name = 'Doe'
ORDER BY f.release_year DESC
LIMIT 20;

Напиши запрос, который выведет топ-10 клиентов, которые оставили в сервисе больше всего денег, но не менее 100 у.е. Запрос должен выводить:

ID и полное имя клиента
сумму всех заказов юзера

(Если нужна подсказка) на основе таблиц: payment, customer

SELECT c.customer_id, c.first_name, c.last_name, SUM(p.amount) AS money
FROM payment AS p
JOIN customer AS c ON c.customer_id = p.customer_id
GROUP BY c.customer_id
HAVING SUM(p.amount) >= 100
ORDER BY money DESC
LIMIT 10;

Общие представления о NoSQL

Что такое NoSQL БД? Какие типы NoSQL БД выделяют? Примеры

Хочется услышать хотя бы самые общие вещи: NoSQL БД — коллекция данных в одном из форматов (см. типы ниже). Данные денормализованы, JOIN реализуют обычно на уровне приложения.

Типы NoSQL БД:

key-value store (реализация hash table: Redis, AeroSpike)
document-store (аналогично key-value, но в качестве значения документ, например, JSON: MongoDB)
wide column store (HBase, Cassandra)
graph database (Neo4j)

Общие представления о Redis

Что такое Redis? Почему его так любят использовать для хранения кэшей? Чем, с точки зрения хранения данных, он принципиально отличается от, например, MySQL? Что произойдет с данными в Redis, если прозойдет рестарт сервера? Какие существуют механизмы для High Availability в Redis?

Ожидания от ответа: Redis – это in-memory key-value БД. У нее есть механизм снапшотов данных из памяти на диск. Для HA используют, например, Redis Sentinel

Тестирование

Общие представления о unit-тестировании

Что это такое?
Чем отличается от интеграционного, функционального тестирования?
Какие библиотеки для Python использовал для этого? (unittest, pytest, в чем разница?)
Когда нужно писать unit-тест? (в существующем проекте, в новом проекте, при починке бага, при реализации фичи)

Как протестировать функцию, которая изменяет запись в MySQL?

Есть функция, которая выполняет INSERT в тестовой базе данных MySQL, которой пользуются все разработчики команды. Нужно написать тест на эту функцию. Но не хочется вносить изменения в общую базу. Как можно решить эту проблему?

mock
transaction rollback

DevOps/Администрирование

Проблема с нечитаемым ответом google.com

Я захожу на https://www.google.com в браузере, открываю исходный код и вижу вполне читаемый текст.

Но если я делаю такой запрос через curl то вижу нечто нечитаемое, похожее на бинарный файл:

> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.58.0
> Accept: */*
> Accept-Encoding: gzip, deflate

%�j��~d+.K�W��me`�,x�x�m�G�67��`$#������~{d���]�.'����
����Q���Mʥu���<�O),�&�����,���E�.���_�
&�I\��p��#]�,�����K+����p�vk
j���k"�fe�=�`�J4�I/��m6{�E���;3+�v�BT1T��(W��J��m�yO��
�j��

Почему так происходит?

Для интервьюера:

curl -v "https://www.google.com" --header "Accept-Encoding: gzip, deflate" --output -

Топ IP адресов из лога NGINX

Наш сервис парсят боты. Это создает большую нагрузку, мешающую нормальным пользователям. Мы решаем забанить ботов по IP-адресам. Для этого мы хотим составить топ IP-адресов, с которых приходили запросы. Мы просим админов сделать выгрузку логов за 1 час.

С помощью стандартных GNU utils создай файл с топом IP-адресов в порядке убывания частоты запросов. Предположим, что IP-адрес находится в 9-й колонке лога, колонки отделяются через t

awk '{print $9}' nginx.log | sort | uniq -c | awk '{print $1","$2}' | sort -r -t, -nk1 > bots.csv

Docker

Что такое докер контейнер?
В чем отличие от virtualbox и других виртуальных машин?
Зачем использовать докер, если можно просто скопировать файлы на машину?
Почему нельзя просто использовать virtualenv?

Архитектура

Parental advisory: highly opinionated content (35+)

Проектирование сервиса копипасты pastebin

Бизнес-описание

Пользователь сабмитит текст
Текст сохраняется, ему присваивается уникальный URL
По переходу по сгенерированному URL мы видим текст
Текст хранится какой-то срок (в минутах)
Сервис удаляет просроченные тексты
Сервис высоконагруженый
Нужна статистика для посещения различных урлов по месяцам

Ограничения и допущения

Пасты только текстовые
Аналитика может быть не реалтаймовой
10 миллионов записей в месяц (3.85 RPS)
100 миллионов чтений в месяц (38.5 RPS)
Железо планируем на 3 года вперед

Ответить соискателю, если спросит

Средний размер пасты 1Kb
Короткая ссылка длиной 7 байт в БД
TTL ссылки в минутах 4 байта в БД
Время создания ссылки 5 байт
Путь хранения файла пасты - 255 байт
Итого: 1.27 Кбайт на 1 пасту

1.27 Гб пасты в месяц За 3 года: - 36 * 12.7 = 457 Гб пасты - 36 * 10 = 360 млн урлов

Крупноблочная схема архитектуры

  Client
    |
    V
Load Balancer
    |
    V
 Web Server (Reverse Proxy, i.e. Nginx)
  |      |
  V      V
Write   Read                                Analytics (MapReduce)
API     API---------------------------       |         |
 |       |        |                  |       |         |
 |       V        |                  |       |         |
 |      Cache     |                  |       |         V
 ------------------------------      |       |       SQL for Analytics
 V                |            |     |       |
SQL-----------    |            V     V       V
 |_Master    |    |           Object (File) Store
 |_Slave     V    |
            SQL   V
 ^              Read-only Replica
 |
 |
Periodic Task
(remove expired)

Write API

Генерит URL, проверяет уникальность в SQL
Если уникальное, делает INSERT в таблицу, иначе идет в п. 1

shortlink char(7) NOT NULL
expiration_in_minutes int NOT NULL
created_at datetime NOT NULL
paste_path varchar(255) NOT NULL
PRIMARY KEY(shortlink)
CREATE INDEX shortlink_idx ON pastes (shortlink) USING BTREE;;
CREATE INDEX created_at_idx ON pastes (created_at) USING BTREE;;

Создание уникального урла

Берем рандомную строку (или IP address юзера + timestamp) и вычисляем MD5
Кодируем Base 62 (как Base 64 но без escape-символов)

def base_encode(num, base=62):
    digits = []
    while num > 0
      remainder = modulo(num, base)
      digits.push(remainder)
      num = divide(num, base)
    digits = digits.reverse

От строки в Base 62 можем взять 7 символов это даст 62**7 возможных значений (это намного больше чем 360 миллионов урлов, которые мы планируем на 3 года)

url = base_encode(md5(ip_address+timestamp))[:URL_LENGTH]

Запрос к Write API

$ curl -X POST https://pastebin.com/api/v1.0/paste \
       --data '{ "expiration_in_minutes": "60", "paste_contents": "My paste" }'

Ответ Write API

{
    "shortlink": "xJlsmeT"
}

Запрос к Read API

$ curl https://pastebin.com/api/v1.0/paste?shortlink=xJlsmeT

Ответ Read API

{
    "paste": "My paste"
    "created_at": "YYYY-MM-DD HH:MM:SS"
    "expiration_in_minutes": "60"
}

Аналитика

Поскольку не нужен реалтайм, будем брать логи с реверс прокси и скартимливать их в MapReduce

class HitCounts(MRJob):

    def extract_url(self, line):
        """Extract the generated url from the log line."""
        ...

    def extract_year_month(self, line):
        """Return the year and month portions of the timestamp."""
        ...

    def mapper(self, _, line):
        """Parse each log line, extract and transform relevant lines.

        Emit key value pairs of the form:

        (2016-01, url0), 1
        (2016-01, url0), 1
        (2016-01, url1), 1
        """
        url = self.extract_url(line)
        period = self.extract_year_month(line)
        yield (period, url), 1

    def reducer(self, key, values):
        """Sum values for each key.

        (2016-01, url0), 2
        (2016-01, url1), 1
        """
        yield key, sum(values)

Полезные ссылки

Toptal Python Interview Questions
Python Interview Questions (junior, middle, senior)
System design interview
The Python Language Reference
PostgreSQL Tutorial

Errata

2020-04-18. В функции проверки сбалансированности скобок исправлена ошибка в маппинге. Для хранения ключей и значений использовались структуры типа set (не гарантируют сохранения порядка вставки элементов). Исправлено на tuple. Нашел ошибку Егор Моренко.

PostgreSQL: update by spliting result set into evenly sized chunks using window function

2019-12-07T14:58:00+01:00

Sometimes you need to process your database data in batches. It may be absolutely necessary to retrieve the rows from the database in your application first, and then process it in the code iterating over the results. In this case you split the result set from the database query into chunks in you code like in the following Python snippet:

from typing import Any, Sequence, Iterator

def get_chunks(sequence: Sequence[Any], chunk_size: int) -> Iterator[Any]:
    for i in range(0, len(sequence), chunk_size):
        yield sequence[i : i + chunk_size]

for ids_chunk in get_chunks(sequence=account_ids_from_db, chunk_size=10):
    # do something with a chunk here
    for account_id in ids_chunk:
        ...

Depending on the number of rows to be retrieved from the database, the SQL query plan and some other factors, you may need to use LIMIT and OFFSET clauses in your SQL and make multiple queries, instead of selecting all the rows at once. It saves you some memory in the application.

But sometimes you may fully rely on the DBMS features to process data in batches. Let’s take a look at the example.

Problem: update blacklist expiration timestamps in chunks

We’ve got a black list of customers. Each item in the blacklist table in the database may have an optional value in expiration timestamp column. If the value is NULL then a client considered permanentely blacklisted. Otherwise a row will be removed from the blacklist table (possibly by a cronjob task), once an expiration timestamp is less than current time.

Problem: we want to set an expiration date to all permanently blacklisted customers so that they get removed from the list in batches of size 10 a day. When splitting into chunks, we want to order the rows so that the customers with the most recent suspicious activity come first. The activity log defined in another database table.

In other words we want to update all rows in the table where some conditions met (expiry value equals to NULL in this case). But we have to do it in chunks of the constant size.

We can do it like in the example above: select all the rows that satisfy the condition, then iterate over batches of size 10 and update. But in this case there’s nothing that a modern RDBMS couldn’t achieve by itself, without writing any extra code. So let’s see how to do it using pure PostgreSQL 9+ query.

Solution: PostgreSQL query for updating rows in chunks using row_number() function

First, let’s see how the table’s schema may look like.

CREATE TABLE balcklist(
  customer_id INTEGER PRIMARY KEY,
  expiry TIMESTAMP WITHOUT TIME ZONE default NULL,
  timestamp TIMESTAMP WITHOUT TIME ZONE default (now() at time zone 'utc')
);

A log of suspicious activity is stored in suspicious_activity database table that we can join using customer_id column.

The algorithm we are going to use when writing final SQL query is the following:

Join activity table, filter and select the data to be updated
Get row numbers using PostgreSQL’s window function row_number()
Get batch numbers using simple arithmetic operations (row number integer division by chunk size)
Get new expiry timestamps using current batch number and PostgreSQL’s interval type arithmetic operations
Wrap up all previous steps into a subquery; update expiry column for customers selected in the subquery.

The final SQL query then is the following:

-- main: update expiry timestamp column in chunks
UPDATE blacklist
SET expiry = subquery.new_expiry_timestamp
FROM (
  -- subquery 3: get new expiration timestamps based on a chunk number
  SELECT
    row_num,
    row_batched,
    customer_id,
    NOW()::TIMESTAMP + row_batched * INTERVAL '1 day' AS new_expiry_timestamp,
    latest_suspicious_activity,
    blacklist_timestamp
  FROM (
    -- subquery 2: get chunk numbers based on a row number
    SELECT
      row_num,
      ((row_num - 1) / 10) + 1 AS row_batched,
      customer_id,
      latest_suspicious_activity,
      blacklist_timestamp
    FROM (
      -- subquery 1: get row numbers
      SELECT
        row_number() OVER (ORDER BY latest_suspicious_activity DESC NULLS LAST) AS row_num,
        customer_id,
        latest_suspicious_activity,
        blacklist_timestamp
      FROM (
        -- subquery 0: order blacklist
        SELECT
          blacklist.customer_id AS customer_id,
          MAX(suspicious_activity.created) AS latest_suspicious_activity,
          blacklist.timestamp AS blacklist_timestamp
        FROM blacklist
        INNER JOIN suspicious_activity ON blacklist.customer_id = suspicious_activity.customer_id
        WHERE blacklist.expiry_timestamp is NULL
        GROUP BY blacklist.customer_id
        ORDER BY latest_suspicious_activity DESC, blacklist_timestamp DESC
       ) AS ordered
    ) AS numbered
  ) AS offsetted
) AS subquery
WHERE blacklist.customer_id = subquery.customer_id

The SQL above may not be SQL-standard compatible and is guaranteed to work on PostgreSQL 9.6 or later.

Comments, labels and the algorithm above should make the query self-explanatory, although some basic knowledge of SQL is needed to understand what the query really does.

Basically, we need only new_expiry_timestamp and customer_id columns from the subquery. But some extra columns are passed from the deepest subquery all the way to the top (e.g. latest_suspicious_activity, row_num, etc), so that you can run the subquery only and see how new expiry timestamps are calculated before doing any updates to the data. BEGIN/ROLLBACK may also be of great help here.

All in all, the query above:

Easy to read
Requires no extra code and does all the batch processing using PostgreSQL query language
Executed much faster than when doing extra processing in your app’s code
Can easily take advantage of PostgreSQL transaction and atomic updates

Switching between Kubernetes clusters and namespaces: kubectx and kubens

2019-09-28T14:12:00+02:00

Having more than one Kubernetes cluster is a common pattern. You may have one cluster dedicated to production environment, another one for staging and the third one for testing environment.

The same thing with Kubernetes namespaces: it’s great abstraction allowing you to separate workloads by their purpose. You may have multiple namespaces for different test environments or a dedicated namespace for app-related pods and another namespace for logging and monitoring services.

Configuring access to multiple Kubernetes clusters is well documented in official k8s documentation. Still switching context and specifying namespace for each kubectl invocation is tedious. This is where kubectx and kubens come into play. But let’s configure multiple clusters first.

Kubernetes configuration file: multiple clusters

You find an extensive guide in the k8s documentation. All in all, your ~/.kube/config should include three lists:

Clusters
Users
Contexts

Context defines a cluster, it’s namespace and a user. When the user switches to the context, he or she is working with the cluster and the namespaces of the context.

Each cluster may define master node’s server hostname and optionaly other entities like base64-encoded certificates.

Users may also be defined by autherntication options that covered in Kubernetes authentication guide.

An example configuration file for two Kubernetes clusters and two users with different authentication types may look like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: YOUR-CERT-AUTHORITY-DATA-CLUSTER-1
    server: https://YOUR-SERVER-ADDRESS-1
  name: cluster1
- cluster:
    certificate-authority-data: YOUR-CERT-AUTHORITY-DATA-CLUSTER-2
    server: https://YOUR-SERVER-ADDRESS-2
  name: cluster2
contexts:
- context:
    cluster: cluster1
    namespace: production
    user: user1
  name: context1
- context:
    cluster: cluster2
    namespace: test
    user: user2
  name: context2
current-context: context1
kind: Config
preferences: {}
users:
- name: user1
  user:
    client-certificate-data: YOUR-CLIENT-CERT-DATA-1
    client-key-data: YOUR-CLIENT-KEY-DATA-1
- name: user2
  user:
    auth-provider:
      config:
        access-token: YOUR-ACCESS-TOKEN
        cmd-args: config config-helper --format=json
        cmd-path: /PATH/TO/google-cloud-sdk/bin/gcloud
        expiry: "2019-09-28T00:00:00Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

kubectx and kubens

Although switching context is easy with kubectl config use-context, working with both contexts and namespaces require way too many key strokes! kubectx and kubens command line tools allow you to switch context this easy:

$ kubectx context2  # the same as kubectl config use-context context2

Chaning context with kubectx changes your config file’s current-context field the same way the kubectl config use-context does.

Now, instead of typing --namespace your-namespace or -n your-namespace in every kubectl invokation, change namespace once with kubens:

$ kubens test2     # your current k8s namespace is test2 now
$ kubectl get po   # the same as invoking with --namespace test2
# kubens test1     # your current k8s namespace is test1 now

Using kubectx and kubens allow you to save some key strokes and really come in handy when you switch over multiple Kubernetes clusters with more than one default namespace.

Application configs: files or environment variables? Actually both!

2019-06-07T20:30:00+02:00

App configuration is highly opinionated topic. It’s technology stack- and infrastructure-dependent. When it comes to configs there are a few parties with a possible conflict of interests. Software developers want an easy access to app’s configuration, versioning, readability, language support for the config format. Security guys want security! SRE/Administrators want an easy deployment and scaling as well as infrastructure native support for configuration format of choice. Managers, well, they probably want to smile at conferences and don’t want you all to spend too much time on such a miserable question like configuration architecture!

In software development some things change rapidly. Ideas, manifests and frameworks rise and fall. So lets review what’s application configuration today and what it could be.

Environment variables: good and bad parts

Environment variables (env vars or env) are often recommended as the only way to store application configs (see The Twelve-Factor App rule III). Envs are good bacause they are:

Secure

Environment variables are set by who’s responsible for the deployment environment: by developer on local machines, by SRE/Administrator/DevOps in production. With env you won’t compromise your production credentials even if your code’s leaked to the public. It’s also harder for developers to break something in production, because they simply don’t have access to production credentials.

Messing up with envs in the runtime may also be tricky, as you cannot change environment variables for the parent process in *nix systems.

Universal

Environment variables are everywhere! On GNU/Linux, MacOS, Windows. Every production-ready programming language supports envs out of box.

Easy to use

Env is flat. Basically it’s a key-value pair with string value. So loading envs is easy.

There are some bad things about envs too. Envs are:

Unexpressive

The flip side of the “easy” coin is unexpressiveness. Envs are flat key-value strings. You cannot store the whole config as en env. So you need to read envs in your code and then construct some sort of the final configuration, possible overly complex.

Flat env structure also get you into a grey area of bad variable names. Consider the following YAML config:

cluster:
  host_weight:
    node00.example.com: 0.8
    node01.example.com: 0.8
    node02.example.com: 1.0
    node03.example.com: 0.5
db: 1
user: app
password: my_password

It may translate to the following envs to substitute values in the config above:

CLUSTER_HOST_NODE_00=0.8
CLUSTER_HOST_NODE_01=0.8
CLUSTER_HOST_NODE_02=1.0
CLUSTER_HOST_NODE_03=0.5
CLUSTER_DB=1
CLUSTER_USER=app
CLUSTER_PASSWORD=my_password

So, you still store some kind of configuration file in your app’s version control system (VCS). Usually it’s a module in you programming language (e.g. config.py in Python) with some language-specific data structures and even some logic. Values (at least some of them) are stored as envs with quite awkward names. These configs are much less readable as the YAML above.

Need to be stored somewhere eventually

Envrironment variables are stored somewhere eventually: in your admins repository, orchestration system’s manifest files that also checked in the VCS. So even separated from the app’s code repo, envs are still stored in files that potentially could be accessed by more people than they should.

May break release determinism

If you use config files checked into your app’s repository, your config is also a part of the app’s releases. You can effectively rollback to a stable release whenever needed.

In the case of envs your app’s releases are separated from the configs. Ideally environment variables should be versioned too. Both app’s and config’s releases should be synchornized somehow. Otherwise your release determinism may break, rollback to previous versions with zero downtime may become difficult.

Configuration files: good and bad parts

Configuration files have their strengths and weakness too. First, the good things. Files are:

Readable

You don’t write config files as program’s module (e.g. config.py in Python). Instead you use some language-agnostic format for humans like YAML. Then you have both readability and (to some extent) separation code and configs.

You write clean, self-sufficient configs. No more ugly things like:

HOSTNAME = os.environ.get('DB_HOSTNAME', 'db.example.com')

Developers often group config files by deployment enviroment: testing.yaml, development.yaml, production.yaml. Sometimes you cannot reach 100% deploy environments parity. So having all configs at hand may help when fixing a bug.

Part of release

As I said earlier, if a config is checked into app’s repository it a part of app release history. It makes your app releases more deterministic allowing to rollback to earlier releases if necessary with no extra efforts.

There are bad things about files too:

Formats hell

XML, YAML, ini, TOML, JSON, you name it! Probably you don’t have a luxuary of having the only one way to do it in your language. Even in JavaScript JSON is not the only configuration format in use. But in popular general-purpose languages like Python it’s even worse.

In a complex project with legacy components you may find all sorts of configuration file formats. It’s annoying.

Insecure

Junior developers, people on probation period, QAs, admins, maybe even developers from other teams and departments may have access to your app’s secrets when they are in the config files checked into a VCS. That’s a huge security problem, even if your network operation center engineers, admins and devops do a great job isolating production services and networks.

Taking good parts from both envs and files

But what if we take the good parts from both enrionment variables and config files, combine them and throw away the bad parts? Sounds great, doesn’t it?

First, we need to take a closer look at what’s really important when it comes to security. Actually, it’s credentials (login/password, private API keys, cryptographic certificates, etc.) that if kept in secret solve most of the problems that your config files stored in the VCS have. If your production database hostname leaks but username and password are kept in secret you are probably okay. Brutforce attacks, unpatched software vulnerabilities are still real. But your customer database isn’t leaked immediately even if bad guys know your database URL.

Second, we really want to have nice language-agnostic non-flat file format for our configs. It’s opinionated, but I personally think that YAML is superior to all other formats. It’s beautiful!

So what if we use a YAML file and substitute our secrets with environment variables tokens?

db:
  login: user
  password: ${DB_PASSWORD}
mail:
  login: user
  password: ${MAIL_PASSWORD:-my_default_password}

Then our app loads the YAML and interpolates environment variables in it:

{'db': {'login': 'user', 'password': 'my_db_password'},
'mail': {'login': 'user', 'password': 'my_default_password'}}

In the above snippet a database password has been loaded from the $DB_PASSWORD environment variable. Email password had to be loaded from $MAIL_PASSWORD. It’s defined as Bash-style env with default value. It seems that at the time of config loading the $MAIL_PASSWORD wasn’t set so a default value 'my_default_password' is used.

Proof of concept

As a proof of concept I’ve just released a Python library called Piny. It does exactly this:

Reads your YAML-config file marked up with environments variables

Interpolates environment variables with their values

Creates a Python object with configuration ready for use in your app

When using Piny consider best practices for your configuration files:

Maintain healthy security/convenience balance for your config

Mark up entity as an environment variable in your YAML if and only if it really is a secret (login/passwords, private API keys, crypto keys, certificates, or maybe DB hostname too? You decide)

Once config loaded by Piny, validate it using your favourite validation tool (some integrations are coming in the future releases)

Store your config files in the version control systems along with you app’s code.

Environment variables are set by whomever is responsible for the deployment. Modern orchestration systems like Kubernetes make it easy to keep envs secure (see Secrets and Vault).

Piny stands for Piny is not YAML. It’s not only a library name, but also a name for YAML marked up with environment variables.

You can download Piny at PyPi.

Helm stable/prometheus-operator: adding new scraping targets and troubleshooting

2019-06-01T14:12:00+02:00

Today I Learned (TIL) that Kubernetes named ports usage may be quite frustrating.

Recently I’ve deployed monitoring stack (Prometheus, Grafana) in Kubernetes using Helm stable/prometheus-operator. Setting up monitoring for Kubernetes cluster itself is covered by this wonderful guide. But for setting up monitoring for other services one need to learn what Kubernetes Operator is, and create his/her own ServiceMonitor for Prometheus-Operator (see stable/prometheus-operator).

ServiceMotitor is a custom resource. It tells Prometheus what k8s Service exposes metrics and where: service label selectors, its namespace, path, port, etc. Say, we’ve got a web application Service in the default namespace with label app: pili that exposes /metrics endpoint on the named port called uwsgi . We are going to deploy ServiceMonitor for Prometheus to scrape the metrics every 15 seconds in monitoring namespace and with certain labels. Then we apply a manifest:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: monitoring-pili
  namespace: monitoring
  labels:
    app: pili-service-monitor
spec:
  selector:
    matchLabels:
      # Target app service
      app: pili
  endpoints:
  - interval: 15s
    path: /metrics
    port: uwsgi
  namespaceSelector:
    matchNames:
    - default

Following this steps I’m sucessfully deployed a few service monitors: for a PostgreSQL cluster, RabbitMQ, ElasticSearch, etc. All of them allowed Prometheus to scrape metrics just as expected. But my own application still showed 0 active targets in Prometheus. I could manually curl my app service’s /metrics endpoint and see that all the metrics are exposed correctly. Still Prometheus was unable to scrape it.

I doublechecked label selectors as they often happen to be a culrpit (see this StackOverflow question). Everything was fine. I could see that my app’s Service exposed Kubernetes endpoints correctly:

$ kubectl get endpoint --namespace default
...
pili  10.244.0.136:8080,10.244.0.144:8080,10.244.0.185:8080  6d16h
...

Eventually, it turned out that ServiceMonitor didn’t see my app service’s named port!

My Service manifest look like this:

apiVersion: v1
kind: Service
metadata:
  name: pili
  labels:
    app: pili
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
    port: 8080
    targetPort: uwsgi
  selector:
    app: pili-web
    tier: backend

In that case Service targeted named port uwsgi from app’s Deployment. The port was also used in Ingress sucessfully:

...
- backend:
    serviceName: pili
    servicePort: uwsgi
...

It wasn’t untill I explicitly named the port (with the same name) in Service that ServiceMonitor could discover my target. So I rewrote my Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: pili
  labels:
    app: pili
spec:
  type: ClusterIP
  ports:
  - name: uwsgi
    protocol: TCP
    port: 8080
    targetPort: uwsgi
  selector:
    app: pili-web
    tier: backend

So the only change that really helped was this one:

   ports:
-  - protocol: TCP
+  - name: uwsgi
+    protocol: TCP
     port: 8080
     targetPort: uwsgi

Service’s port should be explicitly named even if has the same name as target port’s name. ServiceMonitor understands only explicitly named Service ports, although other entities, e.g. an Ingress, work well without explicit naming. Beware!

P.S. At the time of writing I used kubectl v1.14.2 and helm v2.14.0.

Grafana Dashboard for Prometheus official Python client with Flask App metrics

2019-06-01T11:00:00+02:00

I’ve just open-sourced my Grafana dashboard. The dashboard designed for a Flask web application that exposes metrics with my flask_prometheus_metrics Flask extension.

Dashboard aimed at the apps deployed with Kubernetes, although it can be easily tweaked to be infrastructure-agnostic.

12 dasboard’s panels covers the following metrics:

Requests per second per host, endpoint, HTTP method etc.

Latency

Percentiles (latency within which certain percent of requests served)

Number of 4xx, 5xx errors per second

Error count by endpoint

CPU usage per host

Memory usage per host

Open file descriptors per host

App’s uptime per host

App’s version per host

App’s deployment environment (e.g. development, staging, production) per host

Python interpretor version per host

Dashboard also provides variables:

Prometheus datasource

Time interval

Kubernetes pod name

App’s endpoint

HTTP method

Percentile

HTTP status code for errors

The variables allow to change some panels charts grouing, intervals, labels selection.

Usage is easy:

Install flask_prometheus_metrics exporter to your Flask application
Make Prometheus scraping your app’s /metrics endpoint
Import flask-web-app.json at https://<your-grafana-domain>.tld/dashboard/import

If you do not deploy your app in Kubernetes you may need to tweak flask-web-app.json pod labels to meet your needs. Instead of pod you may use instance or hostname or other label name depending on how your Prometheus handle your app’s hostname.

Pelican plugin: word count and reading time statistics

2019-05-28T12:03:00+02:00

I’ve just released a tiny library for Pelican blog platform called word count. It’s a plugin that extends articles and pages with the basic text statistics: number of words and approximate reading time in minutes. You can use it then in your HTML template. My own theme supports it too.

It turned out that there is another plugin called post_stats. It provides even more extensive stats. But I think it’s an overkill for my purposes: I don’t need text readability features, they won’t work for non-English text, they slow down site generation. I also suspect that post_stats is run against all the contents, including non-text ones, which is also not so good for plugin performance.

My word count plugin explicitly runs against text entities (articles, pages, drafts and translations). It calculates basic statistics I need for my blog. And that’s it.

I prefer to add Pelican plugins as git submodule. It makes it easier to git clone the entire blog with all its themes and plugins on the new machine. It’s also make it easier to version plugins and themes you use. Just check out your submodule to the version needed. Check out my article Pelican blog: up and running for more details.

You can donwload (or better git clone) word_count plugin on the GitHub.

Kubernetes Ingress Troubleshooting: Error Obtaining Endpoints for Service

2019-05-26T12:00:00+02:00

Recently I’ve set up an Nginx Ingress Controller on my DigitalOcean Kubernetes cluster. It’s make up of a replica set of pods that run an Nginx web server and watch for Ingress resource deployment. Controller also fires up a LoadBalancer service that routes and balances external traffic to the Nginx pods. In turn Nginx pods route traffic to your app pods in accordance with rules from app’s Ingress manifest. All in all, the whole topology is the following:

The problem is Kubernetes uses quite a few abstractions (Pods, Deployments, Services, Ingress, Roles, etc.) so that it’s easy to make a mistake. The good news is Kubernetes gives you great tools to troubleshoot problems you have bumped into. In my case the first response I’ve got after I set up an Ingress Controller was Nginx’s 503 error code (service temporarily unavailable). Here is how I’ve fixed it.

Kubernetes Nginx Ingress Controller Troubleshooting

Let’s assume we are using Kubernetes Nginx Ingress Controller as there are other implementations too. Its components get deployed into their own Namespace called ingress-nginx.

The first thing you are going to see to find out why a service responds with 503 status code is Nginx logs. Let’s see a list of pods with Nginx:

$ ~ kubectl get pods --namespace ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-5694ccb578-l82hc   1/1     Running   0          21h

Having only a signle pod it’s easier to skim through the logs with kubectl logs. If there were multiple pods it would be much more convenient to have ELK (or EFK) stack running in the cluster.

Looking up the log we see:

$ kubectl logs -f nginx-ingress-controller-5694ccb578-l82hc --namespace ingress-nginx
...
Error obtaining Endpoints for Service "<namespace>/<your-app-service>":
no object matching key "<namespace>/<your-app-service>" in local store
...

Let’s see what endpoints we have:

$ kubectl get endpoints --namespace <namespace>
...
<namespace> <your-app-service>  <none> 21h
...

Indeed, our service have no endpoints. That means that a Service deployed to expose your app’s pods doesn’t actually have a virtual IP address. There are two cases when a service doesn’t have an IP: it’s either headless or you have messed up with label selectors. Check your label selectors carefully! You know what you’re doing when using headless services. So most likely it’s a wrong label name or value that doesn’t match your app’s pods! So was in my own case, by the way.

Once you fixed your labels reapply your app’s service and check endpoints once again:

$ kubectl get endpoints --namespace <namespace>
...
<namespace>  <your-app-service>  10.244.0.136:8080,10.244.0.144:8080,10.244.0.185:8080  21h
...

Now our service exposes three local IP:port pairs of type ClusterIP! ClusterIP is a service type that fits best to the setup with an Ingress Controller and a Load Balancer routing external traffic to it.

Request to a service now returns 200 OK.

Kubernetes Persistent Volumes: How to List and Copy Files and Directories

2019-05-24T20:15:00+02:00

You have created a Kubernetes workload, e.g. Deployment, that uses a PersistentVolume and a PersistentVolumeClaim. It’s provisioned by your cloud platform, say, DigitalOcean. You want to know what space is available to you, you want to browse files and directories on your Volume, and you want to copy something from the Volume to your host machine. How do you do it?

First, find out your pvc’s mountPath. Your data sits there. Second, you can access it from the pod that uses the PersistentVolumeClaim. Fire up a terminal on the pod and use your favourite tools like ls and df to list files or see stats of the volume usage. Just make sure that an image your pod container is using has all the tools you need.

# open bash on the pod
$ kubectl exec -it redis-master-0 bash

# see disk usage stats
# volume is mounted under /data
I have no name!@redis-master-0:/$ df -h
Filesystem                                                                Size  Used Avail Use% Mounted on
overlay                                                                   158G  6.5G  145G   5% /
tmpfs                                                                      64M     0   64M   0% /dev
tmpfs                                                                     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                                                                 158G  6.5G  145G   5% /health
/dev/disk/by-id/scsi-0DO_Volume_pvc-ae31a80d-7b23-11e9-bffb-0ab3124b7b1c  7.9G   36M  7.4G   1% /data
shm                                                                        64M     0   64M   0% /dev/shm
tmpfs                                                                     3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                     3.9G     0  3.9G   0% /proc/acpi
tmpfs                                                                     3.9G     0  3.9G   0% /sys/firmware

# list files
$ ls -l /data
total 24
-rw-r--r-- 1 1001 1001   130 May 20 18:23 appendonly.aof
-rw-r--r-- 1 1001 1001   175 May 20 17:22 dump.rdb
drwxrws--- 2 root 1001 16384 May 20 17:22 lost+found

Third, you can copy files or directories from or to a Kubernetes pod using kubectl cp:

kubectl cp default/redis-master-0:/data/dump.rdb /home/vitaly/Downloads/dump.rdb

See kubectl cp --help for more details.

Helm stable/redis chart: slave unable to connect to master

2019-05-20T19:20:00+02:00

Recently I’ve tried to install stable/redis Helm chart version 8.0.1. I used production-ready values taken from the repo and slightly modified for needs. I used enabled by default clusterisation, but with only one slave node instead of two.

The release deployed sucessfully, but the slave node kept having problem with liveness probe that eventually led to a CrashLoopBackoff. The logs showed that my slave node is “Unable to connect to MASTER”, although I could easily connect to the master from the redis-client node with redis-cli:

$ kubectl run --namespace default redis-client --rm --tty -i --restart='Never' \
  --env REDIS_PASSWORD=my-redis-password \
 --image docker.io/bitnami/redis:5.0.5 -- bash

# redis-cli -h redis-master -a my-redis-password
redis-master:6379>

It turned out that a networkPolicy section in values-production.yaml was a culprit. networkPolicy.enable is set to true, but networkPolicy.allowExternal: true is commented evaluating to false by default. That makes redis nodes only accessible from the pods with a label {{ template "redis.fullname" . }}-client: "true".

So make master accessible to slave you either:

set networkPolicy.enable: false to turn off NetworkPolicy, or
set networkPolicy.allowExternal: true (i.e. by commenting it out in the values-production.yaml)

That’s it. No more failed liveness probes or CrashLoopBackoff. It works.

P.S. I think that production values example in stable should probably be consistent. For example, stable/postgresql production values file contains:

networkPolicy:
  enabled: false
  allowExternal: true

while stable/redis production values file:

networkPolicy:
  enabled: true
  #allowExternal: true

Helm Cheat Sheet: Personal Top of the Little Known Commands and Features

2019-05-20T11:12:00+02:00

Helm is a package manager for Kubernetes. It’s got an outstanding official documentation and tons of tutorials on the internet. Nonetheless Helm has some helpful commands and features that are easily overlooked when reading the documentation. I won’t even try to cover them all, as it’s quite subjective what to count as “helpful”. What I will try to do though is to give a direction on how to find such features by yourself.

CLI Help

Just like kubectl and minikube Helm’s CLI tool follows the same help patterns for commands and subcommands:

$ helm --help
$ helm get --help
$ helm get values --help

Shell Completion

Again, just like kubectl and minikube Helm provides shell commands completion for bash and zsh. See more in help:

$ helm completion --help

Completion helps a lot both everyday interaction with Helm and its CLI exploration.

Commands Aliases

Some Helm commands have aliases. You can see then in the corresponding section in help. As an example these two commands are the same:

$ helm list
$ helm ls

Chart Values

Helm charts usually get initialized with some default values. It’s worth inspect them before installation:

$ helm inspect values stable/postgresql

Sometimes it’s also very helpful to get values from the deployed release, both explicitly set by user and taken from the defaults:

$ helm get values --all <your-release-name>

Chart Versions

Every chart must have a version number. Charts are versioned under version control systems (VSC) and installed from the chart repositories. You can list existing repos with:

$ helm repo list

Don’t forget to update packages information for your repos with:

$ helm repo update

Sometimes you may want to install specific version of chart. List versions first:

$ helm search --versions stable/postgresql

Now install the chart with a --version argument:

$ helm intall --version 3.17.0 stable/postgresql

Chart deletion

If you ommit --name <your-release-name> argument when doing helm install you get a random release name. Release names are reserved even after your release is deleted, so that you can do a rollback.

If you are using a your own release name you can get into trouble when reinstalling chart with the same name. In order to avoid name clash delete your release like so:

$ helm delete --purge <your-release-name>

In this case you won’t be able to do a rollback though.

Emulate Installation

We’ve already covered inspect a little bit. It allows you to see chart’s info, values and a README file. But sometimes you may want to get dirty in chart’s guts (actually, I always do as I don’t really like the idea of installing something I don’t understand in my k8s cluster). In this case you may want to emulate installation with verbose output:

$ helm install --dry-run --debug --version 3.17.0 stable/postgresql

Having chart manifests at your fingertips is a great way to save time and effort browsing for the source code.

Conclusion

Helm’s got almost everything needed to start working with it in its help. It’s consistent and mimics other k8s CLI tools like kubectl and minikube. Take your time to explore the help! I didn’t really try to cover everything here. I just scratched the surface with less known things that you actually could found out by exploring the help.

I’m going to update this post as I find new features worth sharing.

Minikube Cheat Sheet: most helpful commands and features I wish I knew from the start

2019-05-18T19:00:00+02:00

Minikube is a tool for running Kubernetes cluster on your local machine. Recently I’ve posted about Minikube’s ugly parts. Now it’s time for the good parts!

Disclaimer

At the time of writing I used Minikube version 1.0.0

$ minikube version
minikube version: v1.0.0

It seems that Minikube is still under active development. So some things may break for your version. You may also want to check every now and then that you are using the latest version available:

$ minikube update-check

Help

Minikube’s online presence is strange. You can find some mentions in Kubernetes Documentation. You can find a bunch of markdown files with Advanced Topics in Minikube’s GitHub repo. And that’s it about documentation. But Minikube’s help is a great source of knowledge. It’s consistently written for every command and subcommand:

$ minikube --help
$ minikube start --help
$ minikube addons configure --help

Minikube Shell Completion

Minikube has a great CLI. But you can’t really enjoy a CLI without shell commands completion. That’s why Minikube supports commands completion for bash and zsh shells. Again, everything you need is in help already:

$ minikube completion --help

Resources

By default you have got 2 CPU, 2048M RAM, 20G disk allocated for your k8s cluster. It’s okay for the most cases. Still you may need to tweak it depending on what are you doing. Use start arguments to get what you want:

$ minikube start --cpus 4 --memory 4096 --vm-driver kvm2

By now you’ve already got used to using --help, haven’t you?

Configuration

If you experiment with your local k8s cluster a lot, deleting it and starting all over with new settings, you may want to make some settings permanent. They are saved under ~/.minikube/config/config.json. You can edit it manually, but CLI provides a better alternative:

# get list of settings
$ minikube config view

# set cpus; changes will take effect upon cluster delete and start
$ minikube config set cpus 4

Services

Your Kubernetes Service that’s provisioned to be available outside of the cluster’s network can be opened in your default browser from the command line:

$ minikube service <your-service-name>

To see the full list of services do:

$ minikube service list

Monitoring

Minikube’s shipped with a bunch of great addons. You can list them, configure, enable, disable and open up:

$ minikube addons list
$ minikube addons --help

The most helpful addons I use daily to look up cluster’s status are dashboard and heapster. Dashboard is a web UI enabled by default. It’s a great tool for monitoring both your pods (Workloads section) and the whole cluster (Cluster section). Fire up you dashboard with:

$ minikube dashboard

Heapster is Grafana-based monitoring tool. Enable and open it with:

$ minikube addons enable heapster
$ minikube addons open heapster

SSH

When working with Persistent Volumes you may need to do some ops on your cluster’s file system. Use secure shell to do it:

# log in `remotely`
$ minikube ssh

# set `docker` user password
> sudo passwd docker
> exit

Now that you have logged in “remotely” you can create directories to mount your Persistent Volume, copy files, etc.:

# copy directory recursively from minikube cluster to the host machine
$ scp -i $(minikube ssh-key) -r docker@$(minikube ip):/mnt/data/ .

Afterword

As with every complex piece of software in active development Minikube’s experience may be frustrating. Still Minikube is a great tool with a bunch of cool features. I wish I’d read an overview like this one when I first launched Minikube. Hope it will help someone.

Minikube high CPU usage even with no workload on Linux machine

2019-05-18T11:00:00+02:00

Minikube is a tool that allows you to run Kubernetes for local development. It’s adviced to run all over the k8s documentation, in the tech blogs. Still Minikube looks like a piece of software that is not quite ready for a smooth development experience. At the time of writing minikube’s got a bunch of issues. Some of them are open for a long time, hardly reproducible and quite frustrating. CPU usage with no workload to services is one of them.

Recently I’ve bumped into a problem when an idle minikube process consumes up to 100% of CPU. After a while apiserver stops responding making your cluster not really functioning. It turned out I’m not alone. I run minikube on my Lenovo Thinkpad X1 Carbon machine with 16G RAM and 8 cores (i7 8th gen) on board. I use Ubuntu 18.04 with kvm2 VM driver. So I found this high CPU consumption as quite unexpected and annoying. I’ve solved this problem. But as it sometimes happens in the software development, I’m still not sure what exactly helped. So taking into account some black magic, let’s see what witch brew’s ingredients may help.

Before you begin: monitoring

“If you can’t measure it, you can’t improve it”. Before making any tweaks start measuring CPU usage. Altough top tool is cool, it just cannot beat its successor htop. It allows you to tag processes, filter and sort quickly, see processes and threads in a flat and a tree modes.

On GNU/Linux you most likely want to track QEMU processes resource usage.

Minikube’s dashboard plugin is also a great tool for monitoing resources — per pod (Overview section) or for the whole cluster (Cluster section). To use dashboard, enable the addon and open it:

minikube addons enable dashboard
minikube dashboard

Use recommended Virtual Machine drivers

Since I’m on Ubuntu GNU/Linux I went for a kvm2 driver. Get the recommended VM driver for your specific platform.

Follow excellent Arch Linux Wiki KVM article to make sure you have hardware and Linux kernel support for KVM. You may enable virtualisation support in your BIOS and/or load KVM kernel modules needed.

Use latest OS virtualisation libraries, QEMU tools, VM drivers

This is what I believe really fixed my problem. Make sure you:

Update your OS package information:

# in my case of Ubuntu 18.04 I use
sudo apt update

Install the latest libs and tools (like libvirt, qemu), enable systemD services needed as per KVM drivers guide
Install the latest driver

Run Minikube with correct VM drivers and resources

If you had already run minikube with the default virtualbox drivers, remove your cluster with:

# don't forget ``--p <your-profile-name>``
# if you had started minikube with custom profile name
minikube delete

Make sure you start your new cluster with drivers needed and resources allocation that meets your services requirements. In my case I used more memory and cpus than it’s set by default:

minikube start --vm-driver kvm2 --memory 4096 --cpus 4

Use only the addons you really need

Minikube’s shipped with a dozen of addons like dashboard for web UI or heapster for Grafana-based monitoring. They may be cool and useful, but don’t try to enable them all at once.

In my case enabling efk addons has led to a high CPU usage, so that I have to disable it.

Make sure you enabled only the addons you really need:

# get list of addons
minikube addons list

# check for addons in the global config too
# for the case of recreating your cluster
minikube config view

Conclusion

Minikube’s idea is cool. You can really save some money by running k8s cluster on you laptop instead of setting it up in a cloud service for testing. But all the abstractions that Kubernetes introduces, virtualisation with KVM and QEMU on GNU/Linux make Minikube complex. Minikube’s niche (a developer’s local machine) prevents it from being quickly fixed as, say, cloud production-ready solutions that earn their owners piles of money. So let’s hope that magic I’ve posted here will work out for you too. I also hope the issue with high CPU usage will be fixed by someone with proper skills.

Python MRO and Mixin Classes

2019-05-02T13:50:00+02:00

In practice we don’t usually think about how Python’s MRO works. Complex multiple inheritance legitemately considered to be a bad thing. Usually we have very straight forward hierarchies like this:

In [1]: class A:
   ...:     def a(self):
   ...:         return 42
   ...:
   ...:
   ...: class B(A):
   ...:     def b(self):
   ...:         return 43
   ...:

In [2]: b = B()
In [3]: b.b()
Out[3]: 43

In [4]: b.a()
Out[4]: 42

But one application for relatively complex inheritance hierarchies is mixin classes. We can use the following so called diamond inheritance to add some checks to the method do we are interested in.

class Top:
    def __init__(self):
        print('Top class')

    def do(self, value):
        print('Top.do')


class Left(Top):
    def __init__(self):
        print('Left class')
        super().__init__()

    def do(self, value):
        print('Left.do')
        assert isinstance(value, int), 'Left.do expected integer values only'
        # We explicitly return parent's do method.
        # If there was no such return, then Right.do would never be called
        # when Bottom.do() is called
        return super().do(value)


class Right(Top):
    def __init__(self):
        print('Right class')
        super().__init__()

    def do(self, value):
        print('Right.do')
        assert value >= 0, 'Right.do expected positive numbers only'
        return super().do(value)


class Bottom(Left, Right):
    def __init__(self):
        print('Bottom class')
        super().__init__()

    def do(self, value):
        print('Bottom.do')
        return super().do(value)

We can instantiate Top class when no checks in do method needed, we can use class Left when workgin with integers or we can use Right class when working with non-negative numbers. When both checks are needed though, we use the Bottom class.

The initialization of the Bottom class reveals it’s MRO:

In [2]: b = Bottom()
Bottom class
Left class
Right class
Top class

In [3]: Bottom.mro()
Out[3]: [__main__.Bottom, __main__.Left, __main__.Right, __main__.Top, object]

Now let’s see how multiple inheritance works:

In [4]: b.do(10)
Bottom.do
Left.do
Right.do
Top.do

In [5]: b.do(10.5)
Bottom.do
Left.do
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-1eceeaaa850f> in <module>
----> 1 b.do(10.5)

<ipython-input-1-f358c005e4dd> in do(self, value)
     39     def do(self, value):
     40         print('Bottom.do')
---> 41         return super().do(value)
     42

<ipython-input-1-f358c005e4dd> in do(self, value)
     14     def do(self, value):
     15         print('Left.do')
---> 16         assert isinstance(value, int), 'Left.do expected integer values only'
     17         # We explicitly return parent's do method.
     18         # If there was no such return, then Right.do would never be called

AssertionError: Left.do expected integer values only

In [6]: b.do(-10)
Bottom.do
Left.do
Right.do
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-6-f82b823fc4ec> in <module>
----> 1 b.do(-10)

<ipython-input-1-f358c005e4dd> in do(self, value)
     39     def do(self, value):
     40         print('Bottom.do')
---> 41         return super().do(value)
     42

<ipython-input-1-f358c005e4dd> in do(self, value)
     18         # If there was no such return, then Right.do would never be called
     19         # when Bottom.do() is called
---> 20         return super().do(value)
     21
     22

<ipython-input-1-f358c005e4dd> in do(self, value)
     28     def do(self, value):
     29         print('Right.do')
---> 30         assert value >= 0, 'Right.do expected positive numbers only'
     31         return super().do(value)
     32

AssertionError: Right.do expected positive numbers only

See more about the inner workings of the MRO in my article about C3 linearisation algorithm.

Python Method Resolution Order and C3 linearization algorithm

2019-05-02T12:10:00+02:00

Method Resolution Order (MRO) is a order in which methods should be inherited in the case of multiple iheritance. C3 linearization algorithm is how MRO works under the hood since version 2.3.

Wikipedia does a great job explaining the algorithm. It can be reduced to the following steps:

Linearization (i.e. resolution order) is a class itself and a merge of the linearizations of its parents and a list of the parents itself
Linearization of the class with no parents equals to the class itself.
Merge process is done by selecting the first head of the lists which does not appear in the tail of any of the lists. Where head is the first element of the list, and tail is all but first elements of the list. The heads are repeatedly selected and added to the resulting MRO until all the lists are exhausted.
If a head cannot be selected while not all the lists are exhausted merge is impossible to compute due to inconsistent orderings of dependencies in the inheritance hierarchy and no linearization of the original class exists.

Complex multiple inheritance example, courtesy H2power

Consider linearization process for a class K1:

// first, find the linearizations of K1's parents, L(A), L(B), and L(C),
// and merge them with the parent list [A, B, C]
L(K1) := [K1] + merge(L(A), L(B), L(C), [A, B, C])
// class A is a good candidate for the first merge step, because it only
// appears as the head of the first and last lists
       = [K1] + merge([A, O], [B, O], [C, O], [A, B, C])
// class O is not a good candidate for the next merge step, because it also
// appears in the tails of list 2 and 3; but class B is a good candidate
       = [K1, A] + merge([O], [B, O], [C, O], [B, C])
// class C is a good candidate; class O still appears in the tail of list 3
       = [K1, A, B] + merge([O], [O], [C, O], [C])
// finally, class O is a valid candidate, which also exhausts all remaining lists
       = [K1, A, B, C] + merge([O], [O], [O])
       = [K1, A, B, C, O]

So at the higher level a naive implementation of the C3 linearization algorith can be expressed as a simple recursion:

def mro(cls: type) -> List[type]:
  """
  Return a list of classes in order corresponding to Python's MRO.
  """
  result = [cls]

  if not cls.__bases__:
      return result
  else:
      return result + _merge(*[mro(kls) for kls in cls.__bases__], cls.__bases__)

Then _merge repeatedly checks if its lists are exhausted and append appropriate heads to the resulting MRO:

def _merge(*lists) -> list:
  result: List[Optional[type]] = []
  linearizations = DependencyList(*lists)

  while True:
      if linearizations.exhausted:
          return result

      for head in linearizations.heads:
          if head and (head not in linearizations.tails):  # type: ignore
              result.append(head)
              linearizations.remove(head)

              # Once candidate is found, continue iteration
              # from the first element of the list
              break
      else:
          # Loop never broke, no linearization could possibly be found
          raise ValueError('Cannot compute linearization, a cycle found')

In order to hide some lists internals Dependency and DependencyList abstractions are used:

from collections import deque
from itertools import islice
from typing import List, Tuple, Optional


class Dependency(deque):
    @property
    def head(self) -> Optional[type]:
        try:
            return self[0]
        except IndexError:
            return None

    @property
    def tail(self) -> islice:  # type: ignore
        """
        Return islice object, which is suffice for iteration or calling `in`
        """
        try:
            return islice(self, 1, self.__len__())
        except (ValueError, IndexError):
            return islice([], 0, 0)


class DependencyList:
    """
    A class represents list of linearizations (dependencies)

    The last element of DependencyList is a list of parents.
    It's needed  to the merge process preserves the local
    precedence order of direct parent classes.
    """
    def __init__(self, *lists: Tuple[List[type]]) -> None:
        self._lists = [Dependency(i) for i in lists]

    def __contains__(self, item: type) -> bool:
        """
        Return True if any linearization's tail contains an item
        """
        return any([item in l.tail for l in self._lists])  # type: ignore

    def __len__(self):
        size = len(self._lists)
        return (size - 1) if size else 0

    def __repr__(self):
        return self._lists.__repr__()

    @property
    def heads(self) -> List[Optional[type]]:
        return [h.head for h in self._lists]

    @property
    def tails(self) -> 'DependencyList':  # type: ignore
        """
        Return self so that __contains__ could be called

        Used for readability reasons only
        """
        return self

    @property
    def exhausted(self) -> bool:
        """
        Return True if all elements of the lists are exhausted
        """
        return all(map(lambda x: len(x) == 0, self._lists))

    def remove(self, item: Optional[type]) -> None:
        """
        Remove an item from the lists

        Once an item removed from heads, the leftmost elements of the tails
        get promoted to become the new heads.
        """
        for i in self._lists:
            if i and i.head == item:
                i.popleft()

The whole codebase can be found in my c3linear repository. You can install it from the source code or via PyPI:

python setup.py install  # from the source code
pip install c3linear  # from the Cheese Shop

Then just import it and check against Python’s object’s mro method:

from c3linear.mro import mro

class A: pass
class B(A): pass
mro(B) == B.mro()

Take a look at the tests to dive into more complex multiple inheritance examples.

Pelican blog: up and running

2019-05-01T11:30:00+02:00

Pelican is a static site generator written in Python. I use it for my personal blog. Pelican is easy to install, configure and customize with themes and plugins. I was able to set up the blog with my own custom theme and an Open Graph plugin in a few hours.

Before Pelican I also used Jekyll and Octopress. But since my first language is Python and Ruby ecosystem seems to be in a decline, I deciced to stick to a framework where I can express myself as idiomatic as I can. Last but not least, having to write documentation for Python code with Sphinx I also got used to ReStructuredText format, which is supported by Pelican out of the box.

In this post I’m not going repeat the Pelican’s great documentation. Instead I will focus on the step that allows you to:

Set up your blog quickly
Add your own theme to it
Extend Pelican’s basic functionality with a plugin
Automate build and deploy process, and
Host your static site with GitHub Pages under your custom domain

I use Pelican verion 4.0.1, invoke 1.2.0 and ghp-import 0.5.5 in this post.

Install Pelican

Personally I use pyenv for Python version management. I also use pyenv-virtualenv plugin to incorporate virtualenv into it. So my workflow for the Python projects that do not require containerization is the following:

# Create directory for the project, cd to it
$ mkdir my-project && cd my-project

# Set up virtual env with Python version needed
$ pyenv virtualenv 3.7.2 venv

# Activate virtualenv
$ pyenv activate venv

# Initialize git reposotory, add README and basic .gitignore
$ git init
$ touch README.md
$ echo '*~' > .gitignore
$ git commit -m 'Initial commit'

Now that a repository for your project is ready, install pelican package, fix its version and initialize the app:

$ pip install pelican
$ pip freeze | grep pelican > requirements.txt
$ git commit -a -m 'Fix pelican version'
$ pelican-quickstart

All the questions are pretty straightforward. One thing I would advise though is to confirm that you do want to automate site generation and publication. This step will create a Makefile and invoke’s tasks.py

Custom Theme

You can find a ton of Pelican themes in the wild. It’s so overwhelmingly huge amount of custom themes that I had just given up on finding a theme that’s just right for me. Instead I’ve build my own custom theme. It’s responsive, lightweight and clean. For rapid development I used Skeleton, which is a modern CSS boilerplate.

Although it’s recommended to install your theme either by copying it to the Pelican’s theme path or by creating a symlink, I used git submodule add theme in my blog repo directory. Then I added a path to the theme under THEME variable in pelicanconf.py:

$ cd my-project
$ git submodule add https://github.com/pilosus/pilosus-pelican-theme.git themes/pilosus-pelican-theme
$ echo "THEME = 'themes/pilosus-pelican-theme'" >> pelicanconf.py

Now my custom theme is versioned and can be easily developed separately from the blog. The blog may be generated with whatever theme version I like. I will only need to checkout my submodule to the version I need.

Plugins

The first thing I was disappointed in Pelican’s default theme is a lack of Open Graph tags. You cannot just add custom metadata to your content files and use it in the theme. The easiest way to tackle this problem is plugins.

I derived my Open Graph plugin from this one by tweaking the things I didn’t like. Say, the original plugin gets og:image by parsing images from the rendered HTML content, which is strange, ineffective and will certainly produce low-quality results in many cases. That’s why I decided to develop my own plugin.

Again, I find installing plugin as a git submodule a great way to keep the code versioned, site deployments deterministic and maintanence predictable and easy.

I place plugins under plugins directory in my blog repo, then define PLUGIN_PATHS and PLUGINS variables in the settings:

$ cd my-project
$ git submodule add https://github.com/pilosus/pilosus_pelican_og plugins/pilosus_pelican_og
$ echo "PLUGIN_PATHS = ['plugins']" >> pelicanconf.py
$ echo "PLUGINS = ['pilosus_pelican_og',]" >> pelicanconf.py

Build and Deploy Automation

Our goal is to generate a static site, that can be pushed and served by the GitHub Pages with custom domain support. To make this process less tedious some automation is essential.

If you have followed an advice in the Install Pelican section to opt for automation, then you’ve got tasks.py and Makefile installed. Although I do use Makefile in some of my projects I decided to give invoke a try. So the following recipe is all about tasks.py that invoke uses.

We need to install invoke package in your virtualenv, as well as ghp-import for GitHub Page push. Don’t forget to fix all dependencies in the requirements.txt:

$ cd my-project
$ pyenv activate venv
$ pip install invoke
$ pip freeze | grep invoke >> requirements.txt
$ pip install ghp-import
$ pip freeze | grep invoke >> requirements.txt

Now that we have all dependencies installed let’s add a new task for GitHub Pages deployment:

@task
def github(c):
  """Publish to GitHub Pages"""
  preview(c)
  cname(c)
  c.run('ghp-import -b {github_pages_branch} '
        '-m {commit_message} '
        '{deploy_path} -p'.format(**CONFIG))
  c.run('git push --force {github_repo} '
        '{github_pages_branch}:{github_repo_target_branch}'.format(**CONFIG))

preview(c) is a predefined invoke task that generates static files for production environment (i.e. with publishconf.py settings file).

cname(c) is a task that generates a file called CNAME with a hostname of your custom domain:

@task
def cname(c):
  """Generate CNAME file with you custom domain name

  Its used in GitHub Pages. Otherwise custom domain name setting
  gets reset on each git push to GH Page repo.
  """
  c.run('echo {custom_domain_name} > {deploy_path}/CNAME'.format(**CONFIG))

cname docstring says it all. You really need this file!

ghp-import command checks out you output directory (i.e. the one used for generated static files) to the GitHub Pages branch of your project. Then git push --force pushes this branch to the repository for static GitHub Pages (it should be named as your-github-login.github.io).

Configuration you use throughout the invoke’s tasks.py may look like this:

CONFIG = {
    # Local path to content directory
    'content_path': 'content',
    # Local path configuration (can be absolute or relative to tasks.py)
    'deploy_path': 'output',
    # Github Pages configuration
    'github_pages_branch': 'gh-pages',
    'github_repo_target_branch': 'master',
    'github_repo': 'git@github.com:pilosus/pilosus.github.io.git',
    'custom_domain_name': 'blog.pilosus.org',
    'commit_message': "'Publish site on {}'".format(datetime.date.today().isoformat()),
    # Port for `serve`
    'port': 8080,
}

Keeping two separate repositories (one for the blog and another one for the static files it generates) allows you to make your blog repository private. You can keep some secrets or some code you don’t want to share. But still your generated files will be accessible to others in the second repository.

Pelican Workflow

Now that we have defined all the invoke tasks, we can discuss Pelican’s workflow. It’s very straight forward:

Write your content
Rebuild the site with development settings: invoke rebuild
Serve website at localhost: invoke serve
Go to http://localhost:8000 check if everything is okay
Clean output directory: invoke clean
Build website with production settings and upload to the GitHub Pages: invoke github
Enjoy your blog!

Custom Domain

By now you already have your static site live on your-github-login.github.io. In order to serve it under custom domain you need to add a CNAME record in your DNS and set custom domain in the settings of your-github-login.github.io repository on GitHub. Enforce HTTPS is also a great option to turn on.

What’s next?

Althouh the set up and automation I described here are pretty convenient, one thing could still be improved. We could use Travis CI or Circle CI to generate static files and push to the proper repository. CI pipeline should be triggered on each push to the remote blog repository.

Method vs. Class Method Quirks in Python 3

2019-04-29T17:30:00+02:00

A code snippet in Python 3 below has been discussed recently on the internet:

In [1]: class A:
   ...:     def method(self):
   ...:         pass
   ...:
   ...: class B:
   ...:     @classmethod
   ...:     def method(cls):
   ...:         pass


In [2]: A.method is A.method
Out[2]: True

In [3]: B.method is B.method
Out[3]: False

The question is why A.method returns the same object on each call while B.method returns the new one?

Let’s explore first what A.method really is:

In [4]: A.method
Out[4]: <function __main__.A.method(self)>

In [5]: type(A.method)
Out[5]: function

In [6]: import inspect

In [7]: inspect.ismethod(A.method)
Out[7]: False

Essentially, what that means is that A.method is a function, i.e. unbound method of the class A. A great way to check if a method is indeed unbound is to check if its function object and a class instance exist:

In [13]: hasattr(A.method, '__func__')
Out[13]: False

In [14]: hasattr(A.method, '__self__')
Out[14]: False

See Python documentation for more details, specifically an instance method subsection of the Data model section.

On the other hand, B.method is a class method. It returns an instance of the bound method object. Let’s check it:

In [15]: B.method
Out[15]: <bound method B.method of <class '__main__.B'>>

In [16]: type(B.method)
Out[16]: method

In [17]: inspect.ismethod(B.method)
Out[17]: True

In [18]: hasattr(B.method, '__func__')
Out[18]: True

In [19]: hasattr(B.method, '__self__')
Out[19]: True

In [20]: B.method.__self__ is B
Out[20]: True

In [21]: B.method.__func__
Out[21]: <function __main__.B.method(cls)>

That means on each B.method invocation a new object is created. You can achieve the same behaviour with A.method once the method gets bound:

In [22]: a = A()

In [23]: id(a.method)
Out[23]: 140613204502216

In [24]: id(a.method)
Out[24]: 140613210350280

In [25]: id(a.method)
Out[25]: 140613209057096

In [26]: a.method is a.method
Out[26]: False

Again, every time you call a.method a new bound method object is created.

The example above doesn’t apply to Python 2.7 though. It seems that Python 2.7 had different implementation, so that A.method would return different objects every time it’s called.

Let’s get started!

2019-04-28T16:30:00+02:00

My name is Vitaly R. Samigullin. I’m a software developer, a team leader, and a person who enjoys thinking about how and why things work the way they work. Having lived in Austria and Spain, I got some interesting immigration experience. I’ve also started a small publishing company in 2008, which is run by my family now.

Over the last few years I’ve written a bunch of texts in social media that some people confessed to find though-provoking and interesting. As social networks tend to disurupt the openness of their services and the internet, making user’s content ephemeral and hardly discoverable, I decided to start a personal blog. It’s somewhat 2000-ish. But still a great tool to categorize and back up texts that I consider worth saving.

Although stand-alone blogs never get even a fraction of attention that social networks get, a blog post is still unbeatable when it comes “slow reading”. It’s an essential component of learning, actually for both readers and an author. So I except the blogging to become a tool to learn new things through explaining them to others.

I don’t really want to limit this blog to technology and software development. I enjoy a variety of topics ranging from arts and architecture to literature and philosophy.

I speak a couple of natural languages. With Russian as my first language, I’m also fluent in English and have B1-level German. As every unilingual person I would consider my mother tongue a one-size-fits-all thing. Still there are some fields (like computer science, kayaking or playing bass guitar) that easier to think about in language you used to learn that field. I wouldn’t even try to translate my texts to all the languages I’d mentioned. Rather I will stick to the language that fits best to each post. So expect here a Babylon-flavoured mélange.

How often will I post new posts? Will you find the blog completely abandoned in a couple of years? I have no idea. But I do know that quality is over quantity.

So let’s get started!

Vitaly Samigullin's blog

emacs tramp: ssh + sudo

Unmaintained technical documentation

Ownership & docs form

Docs instead of automation or a process

Multiple WiFi network connections on Rapspberry Pi 5

Headless set-up

Desktop

DNS sinkhole with dnsmasq on local Linux machine

Linux + NetworkManager + dnsmasq

Check if local DNS works as expected

Configuring connection’s settings

dnseen: simple DNS queries analyzer

Goals

Alternatives

dnseen

Findings

Conclusion

Recording a screencast on Linux

My Linux journey so far: status update 2023

GNU Stow repo with git submodules

Example stow directory

My personal dotfiles

Clojure: decision fatigue

Possible decisions

Enriched version of clojure-toolbox.com

Clojurians Slack

Debugging and patching Clojure code running in production using socket server and REPL

Practicle example

Get sample code

Compile the code

Run the app with socket server

Check if app is running properly

Connect to REPL

Access state of the running app

Patch the code of the runnning app

Check if changes applied

Revert changes by restarting the process

Conclusion

Precautions

kairos: crontab parsing library for Clojure

Java interop

Clojure CLI and deps.edn

dienstplan: Slack bot for duty rotations

Usage example

Up & Running

pip-license-checker: detect third-party dependencies with “wrong” licenses

Book review: The Clean Coder by Robert C. Martin

Product quality

Time estimates

Keeping competences and skills up to date

Understanding business goals and interests

Conclusion

PostgreSQL: find sequences of same-parameter values within ordered groups

workflow-tools: start using GitHub Actions at scale

Examples

Generating workflow

Setting secrets

Why workflow-tools?

Who should use workflow-tools?

Project links

Python logging: why printf-style string formatting may be better than f-strings

Python logging optimization: format string until it cannot be avoided

Sentry integration with logging

Conclusions

Updates

Python third-party tools configuration: pyproject.toml vs setup.cfg

6 popular Python tools and their configs

Conclusion

Python interview questions and answers

Вопросы и ответы для собеседования веб-разработчика на Python

Why `workflow-tools`?

Who should use `workflow-tools`?