Hangar Board: the token count so far

In the previous post I wrote about building Hangar Board, the flying-group app, with Codex doing the coding. Here is the usage so far:

 

Hangar Board: writing a flying group app with Codex, without looking at the code

I've been having some more spare time recently. Since I have been using coding agents at work for some time now, and in recent months have switched mainly to Codex, I wanted to see whether I could write a fully functional app from scratch without writing a single line of code and without looking at the code whatsoever.

I own a share in an airplane. Our group uses a rather old piece of software for bookings. It does its job, but... it feels very much like old software. So the idea was simple: build something more modern for our group, but do not force everyone to move at once.

For the time being, bookings in the new app work as a wrapper around the existing booking service. The old system remains the source of truth, and anyone who wants to keep using it directly can still do so. The new app just gives us a nicer, group-specific front end and adds the things around bookings that we actually care about.

Current status

The app is called Hangar Board. At this point it is no longer just a toy experiment. It is fully working.

The main pieces are:

• Bookings - view existing bookings, make new ones, amend them, cancel them, etc. There is a compact calendar which makes availability much easier to read and detects conflicts before submitting changes.
• Aircraft page - shows useful aircraft status in one place, including cached legacy booking-system aircraft details, serviceability, check dates, latest observed airport, latest flight, and reference speeds/limits.
• Aircraft tracking - integration with a third-party flight-tracking service whose data sources include terrestrial and space-based ADS-B. The returned flights and track-coordinate payloads are stored in a local database so users can view tracks on a map, get basic stats, export tracks in a format that can be loaded into Google Earth, and look back beyond the most recent live response window.
• Aviation weather - METAR and TAF for our home airfield, or any other airfield we want to query. This is another third-party integration.
• Documents - our group documents are already in a shared Dropbox folder, and I did not want to change that. Hangar Board integrates with Dropbox so users can view or download documents, while local copies can also be used by analytics and AI features.
• Notifications - WhatsApp and email integration for booking notifications, with users able to opt in.
• Checklists - checklists for different stages of operating the aircraft. This is implemented as an installable, offline-capable PWA using a web app manifest and service worker caching. On an iPad or iPhone it looks and behaves close enough to a native app: select the checklist, tick off the steps, and it still works without internet access.
• AI Hangar - a chatbot using the ChatGPT Codex backend, with some of our group documents and the POH available as context. Users can ask questions about those documents, and can also use AI to prepare or modify actual bookings through the normal booking dialogs.
• Roles - app admin, user admin, normal user, and limited user accounts for people who only need restricted access, such as maintenance users.
• Other small things - calendar feeds, light/dark theme support, analytics, admin health checks, logs, backups, and the usual bits that turn an experiment into something people can actually use.

Integrations

One of the most interesting parts of this experiment was that all of the third-party integration work was done by Codex. Aviation weather, aircraft tracking, maps, Dropbox, WhatsApp, email - Codex suggested which services to use, went and read the API documentation, and then wrote the integration code.

My role was basically to choose from the options it recommended. I did not read the API specs. I did not write the request/response handling. I did not wire the service layers. I just guided the product decisions and checked whether the resulting app behaved correctly.

Testing the UI

Another part that still feels slightly magical is that Codex could test the UI on devices by controlling Xcode simulators. It would start a simulator, open a browser, navigate the app, take screenshots, inspect what was wrong, adjust the UI, and repeat.

This was especially useful for the checklist PWA and iPad layouts. I did not have to manually resize windows, copy screenshots around, or describe every alignment issue. Codex could see enough of the rendered result to iterate.

Tests

Codex also wrote the test suite. There are now over 800 tests, and Codex runs them every time a code change is made. That part matters a lot, because once the app grew beyond a few simple pages, manual checking would not have been enough.

Workflow

For this work I have been exclusively using GPT-5.5 xhigh. The whole process has also been a lot of fun. Sometimes I was not even at my computer: I would use ChatGPT with remote Codex access, ask Codex to implement something, and then come back later to review the result. That still feels really cool.

Result

The app is still deliberately small-scale. It uses SQLite and a dependency-light Python web app. It probably would not scale well for some larger deployment, but that is not the problem I am solving. There are six of us. For that, it works very well.

The main goal has been achieved: I did not write a single line of code, and I did not read the code. The app is fully working and working well.

Frankly, although I did expect Codex to succeed, I also expected that I would need to intervene in small ways here and there. I thought I would need to fix some code manually, read a few awkward parts, or step in when an integration got messy. That did not happen.

That said, I do think my technical background helped. I did not write the code, and I did not read the code, but knowing how software, backends, APIs, authentication, persistence, testing, and deployment usually fit together made it much easier to ask the right questions, judge trade-offs, and notice when something needed another iteration.

Still, it feels like we are now very close to the point where an AI system can generate a fully working service like this for someone without that kind of technical background. Not a toy demo, but a real, useful, integrated application built around the way a small group actually works.

When I think about it, just several months ago this was not possible. And yet now it already feels natural, almost mundane, and I find myself taking it for granted.

Truly marvelous times we live in.

Full Dynticks Is Not "No Ticks Ever"

On a low-latency Linux host, it is easy to assume that nohz_full (full dynticks / adaptive ticks) means "this CPU will not get ticks." In practice, tick behavior depends on both idle policy and runqueue state.

This post gives a brief summary of how this works on a Rocky Linux 9.x / 5.14.x kernel, with easy-to-reproduce examples.

TL;DR

• nohz_full + CPU isolation does not mean zero ticks in all states.
• With idle=poll, an idle isolated CPU can still take the periodic tick (~HZ, often 1000 Hz).
• On CFS, if an isolated CPU has exactly one runnable task, the scheduler tick can be stopped.
• Once that CPU has 2+ runnable CFS tasks, tick dependency is re-enabled and periodic tick resumes.

Test Setup

... isolcpus=nohz,domain,managed_irq,1-63 nohz_full=1-63 rcu_nocbs=1-63 idle=poll processor.max_cstate=0 ...

Key points:

• CPUs 1-63 are isolated/full-dynticks candidates.
• idle=poll is enabled.
• HZ=1000.

Observation 1: Idle Isolated CPUs Still Tick with idle=poll

When CPUs 2 and 5 were idle, this probe showed about 5000 events per CPU in 5 seconds (~1000 Hz):

bpftrace -e 'kprobe:update_process_times /cpu==5 || cpu==2/ { @ticks[cpu]++; }

             interval:s:5 { print(@ticks); clear(@ticks); }'

@ticks[5]: 4999

@ticks[2]: 4999

...

Why: idle=poll forces the polling idle path, which restarts/keeps the tick while idle instead of entering deeper idle states.

Observation 2: One Spinning Task on CPU 5 Stops Scheduler Tick

A single userspace busy loop pinned to CPU 5:

taskset -c 5 sh -c 'while true; do :; done'

CPU was 100% user, and /proc/<pid>/schedstat showed:

• field 1 (CPU runtime) increasing,
• field 2 (runqueue wait time) not increasing,
• field 3 (timeslice count for this task) not increasing.

(Documentation/scheduler/sched-stats.rst defines field 3 as "# of timeslices run on this cpu". In practice this is a useful proxy for "how often the task gets sliced/re-scheduled".)

Key proof that the periodic tick stopped on CPU 5 with one runnable task: update_process_times drops from ~1000 Hz to 0.

bpftrace -e 'kprobe:update_process_times /cpu == 5/ { @ticks++; }

             interval:s:5 { printf("CPU5 ticks in last 5s: %lld\n", (int64)@ticks); clear(@ticks); }'

CPU5 ticks in last 5s: 4999

CPU5 ticks in last 5s: 5000

CPU5 ticks in last 5s: 3581

CPU5 ticks in last 5s: 0

CPU5 ticks in last 5s: 0

...

Observation 3: Add a Second Spinning Task on CPU 5, Tick Comes Back

After starting a second pinned spinner on CPU 5, /proc/<pid>/schedstat for the first task started showing:

• runqueue wait time increasing,
• timeslice count increasing.

With the same probe still running, update_process_times on CPU 5 went back to ~1000 Hz while both were runnable.

Mechanically (CFS path): when CFS runqueue depth grows above 1 (rq->cfs.h_nr_running > 1 on this Rocky 5.14 host), the scheduler updates tick dependency and sets TICK_DEP_BIT_SCHED, which prevents full-dynticks tick suppression and kicks the CPU back into periodic tick mode. In some newer kernels/trees, you may see equivalent logic expressed with h_nr_queued instead.

This matches scheduler behavior for CFS: with more than one runnable task, periodic preemption/accounting is needed.

Why Low-Latency Systems Use idle=poll

idle=poll is used for determinism, not efficiency:

• avoids deep C-state entry/exit latency,
• reduces wakeup jitter variance,
• keeps cores immediately responsive for bursty workloads.

Scheduling Class Caveats

The "0/1 runnable task => stop tick, 2+ => restart" rule is mainly CFS behavior.

• SCHED_OTHER/CFS: 2+ runnable tasks generally require tick.
• SCHED_FIFO: does not time-slice equal-priority tasks; one can run indefinitely until block/yield/preempted by higher-priority work. Tick may still be forced by other dependencies.
• SCHED_RR: more than one RR task needs tick for RR semantics.
• SCHED_DEADLINE: tick is generally required.

Also, independent tick dependencies (perf events, POSIX CPU timers, RCU, etc.) can force tick even with one runnable task.

Practical Takeaway

If you observe periodic ticks on an "isolated nohz_full" CPU, first ask:

1. Is idle=poll enabled?
2. Is the CPU idle right now?
3. Does it currently have 1 runnable task or 2+?
4. Are there extra tick dependencies active?

In other words, full dynticks is conditional behavior, not a blanket guarantee.

A handy ONTAP REST fallback: POST /api/private/cli with input

The /api/private/cli family is useful, but the “standard”/structured passthrough style (where the CLI maps onto /api/private/cli/<command-path> and parameters become fields/query params) only covers a subset of CLI commands and behaviours.

When that structured approach doesn’t work (or there’s simply no corresponding REST endpoint you can use), there’s a more free-form option: POST /api/private/cli with a JSON body containing an input string. In effect, you hand ONTAP a whole CLI command line to execute. The trade-off is that you typically get CLI-style text output back (not nice JSON records), but it can still be extremely handy for one-off automation and troubleshooting.

Example (raw CLI passthrough via input)

curl -u admin -k -X POST 'https://<cluster>/api/private/cli' \
  -H 'accept: application/json' \
  -H 'content-type: application/json' \
  --data '{"input": "system node run -node * -command \"priv set diag; wafl scan speed\""}'

Enter host password for user 'admin':
{
  "output": "2 entries were acted on.\n\nNode: XXX-01\nWarning: These diagnostic commands are for use by NetApp\n         personnel only.\nWAFL scan speed is 1300\n\nNode: XXX-02\nWarning: These diagnostic commands are for use by NetApp\n         personnel only.\nWAFL scan speed is 1300\n\n"
}

btw: The CONTAP-458497 mentiones the input approach as well.

Consistent -B Buffering in bpftrace

The -B option in bpftrace (none / line / full) now applies not only to terminal output but also when writing to files with -o.

This ensures the same predictable flushing behavior everywhere, making it easier to use tools like tail -f on trace files.

Change landed in PR #4637.

Using Codex with Your OpenAI API Key - CLI & IDE Setup

Codex CLI Setup

1. Install Codex from https://github.com/openai/codex

2. Edit the ~/.codex/config.toml file and add:

 preferred_auth_method="apikey" 

 model_reasoning_effort="high"

Setting the  model_reasoning_effort="high" is optional but recommended.

Codex IDE Setup

On the machine that will access the git repositories (for example, if VS Code runs on Windows but connects to a Linux host via SSH, create the files on the Linux host).

1. Create ~/.codex/config.toml with the same contents as above,

2. Create ~/.codex/auth.json with:

{

  "OPENAI_API_KEY": "apikey"

}

If `auth.json` already exists, remove all other entries and keep only the API key.

3. Install the Codex IDE extension in VS Code (or restart VS Code if already installed).

Provisioning Multiple Users with Unique API Keys

 To provision multiple users, each with a different API key, without having to actually invite the users to your org:

• Create a new project named Codex in your OpenAI settings at http://platform.openai.com. 
• In the project settings, navigate to API Keys → Create new secret key → Service account.
• Set the Service Key Name to codex-USER, where USER is a unique identifier for each user. 

This approach allows the OpenAI dashboard to display Codex/API usage statistics per user.

HAL

In early February 2023, I built a Slack bot to make OpenAI LLMs readily accessible within our company for experimentation. It wasn’t planned, budgeted, or part of any official initiative—just something I hacked together out of excitement for large language models. I called it HAL, after the computer in 2001: A Space Odyssey. What started as a fun experiment quickly became a core tool that many of us now use every single day. HAL originally ran on OpenAI’s GPT-3.5-turbo model, and since then we’ve been quick to adopt every major new model release as they’ve become available. After that initial development to get core functionality working and deployed to all employees, HAL has been extended organically and mostly in my spare time as a hobby—guided by user feedback, the release of new models and features, and whatever seemed fun or useful to implement. 

Why Slack?

Slack was a natural choice. It was already the company’s preferred communication platform, and by embedding HAL there:

• Everyone could use it instantly on desktop, web, and mobile.
• User identification, auditing, and access controls worked out of the box.
• It felt familiar from day one, no training required.
• It allowed us to move fast, deliver something that worked almost immediately, and required minimal ongoing support.
• Running the HAL code on-prem also makes it easier to integrate with internal systems and gives us full flexibility to customize it exactly the way we want.

Fun fact: our CEO, David, was one of the first to try HAL—and of course, one of his very first requests was to ask it to open the bay door. True to its 2001: A Space Odyssey roots, HAL politely refused.

Ways to Interact with HAL

The main way people interact with HAL is through direct messages. The following Slack features are also utilised by HAL:

• Threaded conversations – Each thread acts like its own private HAL session.
• Channels – HAL can join public or private channels. It only responds when mentioned, unless the channel is dedicated to HAL, in which case it behaves like a DM (optionally with multiple participants).
• File uploads – Users drag and drop code snippets, PDFs, or API specs into HAL chats and then ask questions directly about the content.
• Formatted replies – HAL returns its answers as Slack messages using markdown and emojis, supporting inline URLs, code blocks for short snippets, and other pre-formatted output.
• File outputs – For longer replies with code or structured text, HAL uploads them to the channel as files. Slack automatically applies syntax highlighting for supported languages, making the output easy to read, share, and reuse.
• Channel-based personalization – Different channels can run different models or settings. For example, I use lightweight models for quick questions and heavier models like GPT-5 (reasoning high) or o3-pro in coding channels.
• HAL CLI – There’s also an experimental command-line version of HAL, allowing interaction directly from a terminal session for those who prefer working outside of Slack.

One underrated feature is HAL’s ability to summarise recent discussions in a channel—perfect for catching up after being away.

Under the Hood: LLMs

While HAL started with OpenAI models, it was designed to be model-agnostic. Over time, we’ve run it against on-prem LLaMA, DeepSeek, and soon other models like Claude. HAL itself is written in Python and built on the Slack Bolt API, while the HAL CLI is written in Go for easy distribution as a single binary.

Currently:

• Default model – GPT-5 (reasoning: low), so that by default responses are quick. Since the release of GPT-5 we are still experimenting with which model works best as the default.
• Model routing – HAL can automatically consult with a stronger model for certain queries (see details on Spock below).
• Switching logic – HAL dynamically routes requests between OpenAI direct and Azure endpoints. On Azure we provision each model in multiple regions, and HAL automatically picks the fastest endpoint available for a given model.
• Per-channel models – Users can select different models per channel and set various parameters (such as verbosity or reasoning effort) to match their use case.
• Chat history management – All conversation histories are fully managed by HAL itself. It preserves encrypted tokens, injects full usernames and timestamps into messages, and handles continuity. This approach has proven the most flexible, avoids lock-in to any specific solution, and makes it easier to integrate with other LLM providers. When HAL was first built, LangChain wasn’t ready yet—later, there was never a real need to add it.
• JSON mode – HAL currently requests models to reply in JSON with a provided schema (OpenAI's JSON mode), which made implementing plugins far simpler. It also has logic to retry a query a few times if the model responds with invalid JSON or something that doesn’t match the schema—usually prompting the model to fix it itself. (More recent models rarely respond with invalid JSON.). At some point, I plan to switch to Structured Outputs.

Plugins

HAL has a concept of plugins that add additional capabilities beyond standard chat. Some are enabled by default, while others are opt-in depending on user needs. Here are some of the main ones:

• Spock – Our “power mode.” HAL calls a stronger model (GPT-5 reasoning high or o3-pro) with code execution + web search for tough science, math, and programming tasks.
• Web Search – GPT WebSearch or Google Search for real-time info, and for any additional information not already part of the model.
• Download URL – Pulls web page contents (including some internal sites) into the conversation context.
• Confluence Search – Allows HAL to retrieve and use selected internal Confluence pages for context.
• Image generation/editing – From DALL·E to GPT-image-1.
• Email – HAL can send a response directly to a user’s inbox.
• TTS (Text-to-Speech) – Generates an audio file of a response using OpenAI’s TTS models and uploads it to Slack.
• LaTeX rendering – Converts math into images (useful for equations).
• VDI ADM – Allows users to perform actions like reboot, reset, and similar operations on their VDI workstations.

Other plugins have been more experimental, such as Wolfram Alpha or PagerDuty integration, but they’ve demonstrated just how flexible the architecture is.

Three Notable Extensions

1. Spock

Spock is HAL’s “specialist mode.” It uses the heaviest model available, with web + code execution, to tackle deep technical queries. HAL auto-routes to Spock when users ask about physics, biology, or advanced coding—but users can also explicitly request it.

Yes, the answers may take minutes, but they’re often worth the wait.

2. VDI ADM

HAL can also help employees manage their virtual desktops. It allows users to perform actions like reboot, reset, and similar operations on their VDI workstations.

This feature has greatly reduced the number of callouts to our Helpdesk for requests to restart hung VDI workstations (researchers always find a way to hang theirs while experimenting). It’s also very easy for users to remember—no need to learn the name or syntax of yet another tool; you just ask HAL in whatever language you like, even from your mobile phone while away from your desk, and it handles it.

A humorous anecdote: early on, someone asked HAL if it could restart a specific application on their workstation. HAL responded that it couldn’t—but since it could restart the entire workstation (which would restart the application too), it went ahead and did just that. While the potential for this behaviour was known, it hasn’t actually happened for quite some time. Since that incident, the plugin was modified so that it now requires the user to confirm the requested action before it will be executed—HAL can no longer run the action without explicit confirmation by the user.

3. Confluence Search

HAL can also search and process selected internal Confluence pages. This works in two ways:

• Force-feeding technique – A given documentation dump is exposed to HAL as a plugin/function it can call. When triggered, the entire dataset (with some metadata and URLs) is preloaded into a model with a large context window, along with instructions and the user query. HAL is instructed when to use it and can retrieve or synthesize information based on the plugin's reply without polluting the main chat history.

• Download URL approach – For certain topics, HAL is instructed to fetch relevant Confluence pages directly at query time, pulling them into the conversation context to answer questions or for analysis or summarisation.

Together these techniques make Confluence search surprisingly effective, even without embedding databases (which we'll most likely add in the future).

Cost vs. Value

Despite usage growing, our actual cost of using HAL (API calls) has been going down. Model prices are dropping faster than usage is rising, and features like input token caching help further. Given the productivity boost HAL brings, the costs are negligible compared to the value delivered.

Looking Ahead

The LLM landscape is moving at lightning speed. On HAL’s roadmap:

• Support for MCP servers.
• Wider integration with non-OpenAI models.
• Deeper coding integrations.
• User customisations – the ability for users to provide their own additional instructions for HAL to follow, making interactions more tailored to their style or needs.

At some point, we’ll most likely adopt an off-the-shelf solution that will replace HAL—but in the meantime, developing and using it has been a lot of fun, and an incredible way to learn about LLMs and how to get the best out of them.

Recent Updates (Sep '25)

HAL has recently been extended with several new features:

• Anthropic/Claude models added, broadening the choice of LLMs available.

• User customisations introduced, allowing individuals to add their own instructions—either per channel or globally.

• On-call lookup plugin added, enabling users to query who is currently on-call from different internal teams.

• Employee directory plugin introduced, so users can query HAL for the contact details of other employees.

Lessons Learned

While having an LLM assistant directly in Slack is extremely convenient, it also comes with significant limitations. Some of the more notable ones include:

• Message size limits – Longer replies must be split into multiple messages, and Slack formatting often breaks when messages are split. This can be partially worked around in code, but it adds complexity.
• No “thinking” indicator – Slack’s old “user is typing…” signal isn’t supported by newer APIs. Injecting emojis or similar tricks isn’t the same.
• No streaming support – Users have to wait for an entire reply to be generated before they see anything, unlike in a dedicated client.
• Limited Slack markdown – Not rich enough to render more complex messages.
• File size constraints – File snippets have limits; larger files redirect to a browser.
• Basic interactivity – Interactive buttons and message support are limited, with little control over how they work on mobile or iPad.
• File upload limits – Maximum of 10 files per upload.
• Missing context signals – Slack doesn’t provide current estimated user location, which could sometimes be useful.
• Many smaller quirks – Dozens of other small issues that add friction.

Providing an LLM bot via Slack is always going to be constrained compared to a dedicated client. However, if your company already uses Slack, having the LLM available there is still very useful—even if for certain tasks it makes more sense to switch to another tool. In our case, many of us use both HAL in Slack and GitHub Copilot inside VS Code, depending on the situation. Sometimes one is more convenient, sometimes the other.

Our headquarters in London... :)

LLAMA 2 - hello world

 $ torchrun --nproc_per_node 1 test2_text_completion.py \

    --tokenizer_path tokenizer.model --max_seq_len 128 --max_batch_size 4 --ckpt_dir llama-2-7b/ \

    --prompt "Hello everyone, I'm LLAMA-2"

...

Loaded in 4.32 seconds

Hello everyone, I'm LLAMA-2 000. I'm a new member here. I'm from China. I'm a girl, I'm 16. I like animals, especially lamas. I like to play games, especially the games on the internet. I like to make friends. I hope I can make

Azure/DALLE-2

The latest version (0.27.8) of the pypi/openai library now supports Azure/DALLE-2. To ensure compatibility, set openai.api_version = '2023-06-01-preview'.

BPF & Unix timestamp

Recently I've been using bpftrace to trace some events across servers and it would be really useful if bpftrace allowed to use actual unix timestamps. The nsecs variable won't work as it represents time since boot, while time() and strftime() return strings. 

In this specific case, an application running on server A was injecting timestamps into packets being sent to an app on server B where I run tracing. This allowed for tracing time it takes to send a packet over network between apps. Unfortunately bpftrace did not make it easy to do so...

This only makes sense if time across servers is synchronised with accuracy much better that the time deltas you need to measure. In our case, servers use PTP and are generally synchronised to GM <<100ns, while the time differences measured here were in many microseconds.

I'm not the only one asking for actual timestamps in bpf. For example, see here and here. The latter is a discussion about implementing a bpf function to provide epoch timestamp. Unfortunately, there was some resistance in doing so (I don't agree with the arguments), but at least there was a workaround suggested. It is unnecessarily complicated and clanky, but works.

> Not sure what problem you're trying to solve and thus what exactly you
> need... but you can probably get something very very close with 1 or 2
> of clock_gettime(CLOCK_{BOOTTIME,MONOTONIC,REALTIME}) possibly in a
> triple vdso call sandwich and iterated a few times and picking the one
> with smallest delta between 1st and 3rd calls. And then assuming the
> avg of 1st and 3rd matches the 2nd.
> ie.
>
> 5 times do:
> t1[i] = clock_gettime(REALTIME)
> t2[i] = clock_gettime(BOOTTIME)
> t3[i] = clock_gettime(REALTIME)
>
> pick i so t3[i] - t1[i] is smallest
>
> t2[i] is near equivalent to (t1[i] + t3[i]) / 2
>
> which basically gives you a REAL to BOOT offset.

Let's quickly implement it:

#include <time.h>
#include <stdio.h>
#include <stdint.h>

#define ITERATIONS 5

int main(int argc, char **argv) {
  struct timespec ts1[ITERATIONS], ts2[ITERATIONS], ts3[ITERATIONS];
  uint64_t t1, t2, t3, t4, t5, smallest_dt = 0;
  int ret, i, smallest_dt_i;

  for (i = 0; i < ITERATIONS; i++) {
    ret = clock_gettime(CLOCK_REALTIME, &ts1[i]);
    ret = clock_gettime(CLOCK_BOOTTIME, &ts2[i]);
    ret = clock_gettime(CLOCK_REALTIME, &ts3[i]);
  }

  for (i = 0; i < ITERATIONS; i++) {
    t1 = ts1[i].tv_sec * (uint64_t)1000000000 + ts1[i].tv_nsec;
    t3 = ts3[i].tv_sec * (uint64_t)1000000000 + ts3[i].tv_nsec;
    printf("i: %d dt: %lu\n", i, t3-t1);
    if (!smallest_dt || ((t3 - t1) < smallest_dt)) {
      smallest_dt = t3 - t1;
      smallest_dt_i = i;
    }
  }

  t1 = ts1[smallest_dt_i].tv_sec * (uint64_t)1000000000 + ts1[smallest_dt_i].tv_nsec;
  t2 = ts2[smallest_dt_i].tv_sec * (uint64_t)1000000000 + ts2[smallest_dt_i].tv_nsec;
  t3 = ts3[smallest_dt_i].tv_sec * (uint64_t)1000000000 + ts3[smallest_dt_i].tv_nsec;
  t4 = (uint64_t)(t1+t3)/2;
  t5 = t4 - t2;

  printf("\n");
  printf("t5=t4-t2: %lu\n", t5);
}

Now on a system where you need actual timestamp in bpftrace you do: $t_now = (uint64)NNN + nsecs;

where NNN is the t5 timestamp reported for the system.

I wish one could get it directly in bpftrace, but we are where we are...

Siri + ChatGPT

Although I appreciate Siri's ability to automate my home, there are times when it can be frustrating, such as when it replies with the annoying message "I found some web results, I sent them to your iPhone." Let's make Siri much more useful and fun by integrating Siri with OpenAI/ChatGPT via Shortcuts app. It even works with HomePod and Apple Watch.

To get started, we need to define a variable with our OpenAPI token, and then use the Dictate Text action to convert speech to text and store the result in another variable.

Next, let's set another variable with parameters for OpenAI query:

Now, let's issue a REST call:

And finally let's extract the reply and pass it back to Siri to read it aloud.

Now, you can say "Hey Siri, 2001" (I named the shortcut "2001") and it will ask you for a propmpt, once you provide it a moment later it will read the OpenAI/ChatGPT reply to you making Siri really useful and fun to engage with. It even works in CarPlay mode while driving.

You can download the Shortcut as well, enjoy.

btw: I asked ChatGPT to improve the blog post, here is the result:

The given blog post can be improved by making some modifications and clarifications. Below is the improved version of the post:

Siri is an amazing tool that can automate various tasks and make life much easier. However, sometimes it can be frustrating when it responds with the message "I found some web results, I sent them to your iPhone." To make Siri even more useful and fun, we can integrate Siri/Shortcuts with OpenAI/ChatGPT, which can work with HomePod too.

To get started, we need to define a variable with our OpenAI token. Then, we can use the Dictate Text action to convert speech to text and store the result in another variable. This allows Siri to understand our speech and perform the necessary actions.

Next, we need to set another variable with the parameters for the OpenAI query. These parameters include the text we want to generate, the model we want to use, and the temperature of the output. Temperature refers to the randomness of the generated text. Higher temperatures will result in more creative and unexpected output, but lower temperatures will produce more accurate and predictable text.

Once we have set the necessary variables, we can issue a REST call to the OpenAI API. This will send our query to the OpenAI server and retrieve the generated text. We can then use the Text action to speak the generated text aloud or display it on the screen.

It's important to note that using OpenAI/ChatGPT with Siri requires some programming skills and knowledge of APIs. However, with some practice and experimentation, anyone can create their own Siri shortcuts and integrate them with OpenAI. With this powerful combination, we can create a wide range of useful and entertaining applications, from generating creative writing prompts to answering trivia questions.

bpftrace vs. strace

Many years ago I compared performance overhead of dtrace vs. truss on Solaris, in a microbenchmark. This time I run a similar test but comparing bpftrace vs. strace on Linux. To do so, I wrote a simple C program which stawns X threads and each thread will call stat("/tmp") N times in a loop. The program then prints the total time it took for all threads to execute.
While it is not necesarilly a very realistic test, it does show a potential overhead of tracing and difference between different technologies. Also sometimes you do need to trace a very tight loop which then might result in overheads like shown below (or even higher).

Let's run it three times (to see if we get consistent results) with 4 threads, each one calling stat() 100k times.

# for i in $(seq 1 3); do ./threads-loop-test-1 4 1000000; done
Total time: 0.982799
Total time: 0.977269
Total time: 0.980996

It took just below 1s to execute.
Now, let's run it under bpftrace which will count how many times the stat() was called by all threads.

# for i in $(seq 1 3); do bpftrace -qe 't:syscalls:sys_enter_newstat
                                        /pid==cpid/{@[probe]=count();}'
                                   -c "./threads-loop-test-1 4 1000000";
                       done

Total time: 1.192433

@[tracepoint:syscalls:sys_enter_newstat]: 4000019
Total time: 1.204695

@[tracepoint:syscalls:sys_enter_newstat]: 4000019
Total time: 1.143513

@[tracepoint:syscalls:sys_enter_newstat]: 4000019

There is roughly a 20% overhead - not bad.
Adding an extra condition to the predicate str(args->filename)=="/tmp" has little impact - resulting in total times <1.24s.
Again, not bad, especially given that string comparison like this is rather expensive.

Now time for strace.

# for i in $(seq 1 3); do strace -qfc -e trace=stat ./threads-loop-test-1 4 1000000; done
Total time: 49.478656
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00   87.720184          21   4000019        18 stat
------ ----------- ----------- --------- --------- ----------------
100.00   87.720184          21   4000019        18 total

Total time: 49.336942
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00   87.463778          21   4000019        18 stat
------ ----------- ----------- --------- --------- ----------------
100.00   87.463778          21   4000019        18 total

Total time: 49.250562
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00   87.923629          21   4000019        18 stat
------ ----------- ----------- --------- --------- ----------------
100.00   87.923629          21   4000019        18 total

It took about 50 times longer to execute!

While there have been many improvements to strace to reduce its impact, it is still significant in some cases.
It doesn't mean that strace is a bad tool and you should avoid it - in fact, it is often more handy and quicker to use than bpftrace or systemtap.
However be mindful of its potentially much higher overhead, especially in tight loops.


The source code for the test program.

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>

void *thread_handler(void *arg) {
    int N = (long)arg;
    struct stat file_stat;
    while (N--) {
        stat("/tmp", &file_stat);
    }
    pthread_exit(NULL);
}

int main(int argc, char *argv[]) {
    struct timespec start, end;
    int num_threads;
    long N;
    if (argc != 3) {
        printf("Usage: %s  \n", argv[0]);
        exit(-1);
    }
    num_threads = atoi(argv[1]);
    N = abs(atol(argv[2]));
    pthread_t threads[num_threads];
    int rc;
    long t;
    clock_gettime(CLOCK_MONOTONIC, &start);
    for(t=0; t<num_threads; t++){
        rc = pthread_create(&threads[t], NULL, thread_handler, (void *)N);
        if (rc){
            printf("ERROR; return code from pthread_create() is %d\n", rc);
            exit(-1);
        }
    }

 /* Wait for all threads to complete */
    for(t=0; t<num_threads; t++) {
        pthread_join(threads[t], NULL);
    }

    clock_gettime(CLOCK_MONOTONIC, &end);
    double time_taken = (end.tv_sec - start.tv_sec) + (double)(end.tv_nsec - start.tv_nsec) / (double)1000000000;
    printf("Total time: %lf\n", time_taken);
    
    return 0;
}

NFS4ERR_EXPIRED - Linux NFS client bug (fixed)

Another blog entry saved as draft for a long time and waiting to be published... better late than never, so let's do it now.
Over three years ago I found a bug in Linux nfs v4.0 client which results in applications temporarily not being able to open a file and getting an error. While this is due to a bug in the nfs v4.0 client code, depending on a specific nfs server you might or might not hit the issue. For example, both Linux and Solaris NFS servers will trigger the bug, while NetApp/ONTAP won't.

Once I debuged and understood the problem and was able to reproduce it. Then I ended up with a temporary workaround and later submitted two patches to Linux kernel which quickly have been integrated into Linux 5.6 and backported to some older versions (see the patches at the very end of this blog entry).

Let's look in more detail at the issue.

btw: some output has been abbreviated and/or modified to remove irrelevant details or to anonymize details.

An application written in java was sometimes failing to open a file with the below exception:

Caused by: java.io.FileNotFoundException: /.../latest.link (Input/output error) 
        at java.io.FileInputStream.open0(Native Method) 
        at java.io.FileInputStream.open(FileInputStream.java:195) 
        at java.io.FileInputStream.<init>(FileInputStream.java:138) 
        ...

Developers added an retry logic which helped. Usually after few retries the application was able to open the affected file fine. This was happening to different files, at different times on multiple servers. Even to files which were created long time ago (don't get fooled by the java.io.FileNotFoundException).

To understand how a file is accessed by the application (although some other applications were affected too), I wrote the below code based on the affected application.

$ cat Test.java
import java.io.*;
...

public class Test {
    public static void main(String[] args) {
        FileUtils fu = new FileUtils();
        File dir = new File("/mnt/.../indexweights");
        File ft = new File("/tmp/foo");
        try {
            ft = fu.resolveLinkFile(dir);
        } catch (IOException e) {
            System.out.println("IOException: " + e.getMessage());
            e.printStackTrace();
        } finally {
            System.out.println("Destination path: " + ft.getAbsolutePath());
        }
    }
}

Let's run it under strace and see what system calls it calls: 

$ strace -tTfv -o /tmp/a -e trace=file,close,read java Test
Destination path: /mnt/...

$ grep -A2 latest.link /tmp/a
4422  15:33:38 open("/mnt/.../indexweights/latest.link", O_RDONLY) = 5 <0.069349>
4422  15:33:38 read(5, "...", 8192) = 11 <0.000010>
4422  15:33:38 close(5)                 = 0 <0.069149>

It is just a simple open() with O_RDONLY followed by read() and close(). Nothing suspicious. The error is happening with open() though, based on the java stack in the app log.
I did run the test program multiple times on affected servers and tried to access the affected files, but couldn't trigger the issue.

We modified one of the affected applications and added a retry logic for failed file opens, we also run part of the application under strace to see what's the actual error reported by open().  Let's see an example output:

$ egrep ' open\(|open resume' .../XXX.out.20190625220935.26281
...
[pid 133595] 22:10:17 open("/mnt/.../latest.link", O_RDONLY <unfinished ...>
[pid 133595] 22:10:17 <... open resumed> ) = -1 EIO (Input/output error) <0.069362>
...
[pid 133595] 22:10:21 open("/mnt/.../latest.link", O_RDONLY <unfinished ...>
[pid 133595] 22:10:21 <... open resumed> ) = -1 EIO (Input/output error) <0.069204>
...
[pid 133595] 22:10:29 open("/mnt/.../latest.link", O_RDONLY <unfinished ...>
[pid 133595] 22:10:29 <... open resumed> ) = 192 <0.487662>

The open() failed with EIO two times and then succeeded the third time.
Given these are nfs4 mount points with "hard" mount option we shouldn't be getting EIO though.
When the issue happens no errors reported in system logs.
I also collected network traffic for the above failures.