Conflicted about AI

I’m feeling extremely conflicted about AI. On one hand, I’ve been learning about and using AI recently, and I’ve also been pursuing creative pursuits like audiobook narration. I’ve seen many many creative people filled with rage about AI, and specifically people trying to pass off AI-generated works as their own creative output. Even outside of this deception, AI offers “good enough” (highly debatable) versions of art, of music, of speech, and this directly impacts the ability of all the people working in these fields earning money to support themselves, to keep doing what they love and also pay the bills.

George Costanza Me: Was that wrong?

My previous post about AI used a picture that was AI-generated. Immediately following that post I see demigod-from-the-proto-Internet Jamie Zawinski in my Mastodon feed boosting a post declaring what I just did was super bad:

Before you add an AI-generated image to your blog post, have you considered saving even more time and just putting “YOU CAN STOP READING NOW” in 120 point text

Goddamn it. I was so pleased, using these cool new tools and coming up with a prompt that got ChatGPT Sora to roughly give me what I wanted for an image — a (O)llama relaxing at home watching TV in the dark. If I hadn’t done this, it’s not like I would have commissioned a human artist, or tried to draw something myself, I just would have published the post with no image at all and called it good enough.

Isn’t the net addition of an amusing picture to a blog post a good thing? But also, isn’t there something to the negative sentiment?

Also me: This is Not Good.

AI is disrupting the audiobook world. When I, as a person seeking to start doing audiobook narration look to audition for self-published books on Amazon’s ACX platform, the primary way I know people are writing the fiction books is: no AI would write this poorly! But when it comes to nonfiction, it’s much harder to know. Do I need to fall back on the classic LLM tells of bulleted lists and use of em-dashes to guess? How do I feel about auditioning to narrate a book possibly written generated by an LLM? Should it matter? Am I validating or <something>-washing the approach by doing an actual human-produced narration of the book?

At least the book’s author wanted an actual human narration, instead of relying on AI for that too. A human narration is always going to sound better (for now), but when the producer of AI-generated books is using a quantity-over-quality approach to flood the marketplace, will that matter, or will consumers eventually come to accept AI narration as their ears get used to it, much the same way pop music fans are now used to autotuned vocals?

This blog post will eventually be used to train LLMs

LLMs are being trained on the entire accessible internet (sanitized). LLMs are a distillation of all their training, and do not refer back to the original source to respond to a query. There is a negative feedback loop in the making: the more we use LLMs, the less incentive there is to publish on the internet, since it will result in fewer hits by actual users. The less published, the less training data for LLMs, unless they resort to synthetic training data.

I do not believe AGI is coming soon, or that scaling LLMs will result in another quantum leap in their capabilities. But even the current level of technology has only begun to disrupt the status quo. Even if it makes mistakes and hallucinates, it is still incredibly useful, but it will still be years before we understand where best to deploy it. Before then, I think we can expect that someone will try to use AI for every possible application, and some of these will fail in unforeseen and incredibly damaging ways.

So yeah. I’m extremely conflicted.

Local LLMs and Ollama

A Llama relaxing at home.

Lately as part of my time-off learning, I’ve been diving into AI. (Gawd, I know, right?) Of course, they have been such a big deal over the past few years I feel like I’m a little late to the party, but I’m also NOT late to the party. While big proprietary tools like ChatGPT and Claude are no longer new, development is still happening rapidly! Most interesting to me, smaller models, and “open source” models, have improved dramatically, and we can now have our own private LLM instances that we can query, for free and with no data leakage!

Step One: LM Studio on Macbook Air

Download LM Studio. Run it. Download Qwen3 4B Thinking model. Load it. Chat away. How frickin’ easy is this?? I’m running a LLM and don’t need to worry about OpenAI harvesting my queries, or having to pay a subscription. Apple M-series Macs are actually decent at running LLMs, they have a unified CPU/video memory architecture, so a Mac with a lot of RAM can hold and run big models reasonably fast, where you’d otherwise need numerous high-end GPUs with a lot of VRAM, that would be even more money and power.

But on my 16GB M3 Air, after a bit of this, one does start to notice… it’s a little slow, innit? And, the answers, when asking about obscure things, are craaazy. My favorite is to ask about 80s sitcoms My Two Dads or Who’s the Boss. Qwen3 4B said My Two Dads is about a woman named Grace who marries her two fathers after they divorce. Uh, no.

But still, I am hooked!

Hmm, my pandemic-era gaming machine is sitting idle, I bet it could run some more models…

Step Two: LM Studio on Windows with RTX 3090

Installing LM Studio on my Windows gaming machine and downloading some larger models that fit within its 3090’s 24GB VRAM blows my hair back! Better results! And fast! I immediately start making plans to upgrade the system’s RAM, and exploring LM Studio’s server options. Clearly I want a fast and large (or at least medium-sized) LLM: for dumb questions, for smart questions, for programming assistance, which seems to be the number one killer app for LLMs so far. My editor of choice is Emacs (which you will take from my cold, dead hands). I jump through a number of hoops to get LM Studio listening on a port, Windows firewall made aware, and Emacs gptel configured.

Eh, it works ok.

I’ve heard good things about Ollama, let’s try that.

Step 3: Switch to Ollama and Ellama

Ollama on Windows has a GUI, although simpler than LM Studio’s. It still allows you to download models and chat, although that’s about it. The main reasons to use Ollama are: it’s open-source (MIT), it seems to handle model dynamic loading pretty well, and its service API is a little richer than LM Studio’s OpenAI-compatible API.

Alas, trying to copy a 23GB model file from LM Studio to Ollama’s .models/ directory doesn’t work. Looks like some painful re-downloading is in our future.

I ditched gptel and am now using Ellama package, on my laptop running Emacs on Fedora. After some difficulty getting it to connect to the model server, it’s working. Pretty damn slick. It does a lot. A whole lot! I’m going to need to learn how to use all of this. I keep generating text into buffers for source code, and there are a lot of options to explore. Docs exist but is of the “assume you know what everything does” variety, which I still don’t. But, very exciting!

Aside: So Many Models!

Here in mid-2025 the models I’ve been using are gpt-oss:20b, deepseek-r1:32b, gemma3:27b, and llama4:scout. Played some with 120b and 70b versions but they’re so much slower I don’t have the patience. I am also using some proprietary LLMs (ChatGPT, Claude) and they are definitely better, but… I dunno, I just really like the idea of using a local LLM. Smaller models and distillations keep getting better. Maybe I missed the boat on LLMs initially, but I think I started playing with local LLMs at the perfect time.

Really interesting to see where models are coming from. Big US tech firms, and also really good-for-their-size Chinese models. HmmmMMMmmmm.

Step 4: LLM Server RAM upgrade

It had 32GB. I upgrade it to 128GB, the max the motherboard supports. It lets me run larger (e.g. 70B) models, although a bit more slowly, even with Ollama putting as much as it can into the 3090’s VRAM.

One default behavior of Ollama is it likes to unload models after a couple of minutes. This affects initial queries, since it has to load the 23GB file off storage and into VRAM before it can run the LLM. Disable this – keep the last LLM model loaded indefinitely.

It would also be cool if Ollama could keep multiple models loaded in RAM, although maybe only one could use VRAM at a time. Still working on figuring this out.

Step 5: Browser support

I install a Firefox plugin called “Ollama Client” on all my machines. (Does it really only have 30 users??? Is no one else using local LLMs yet??) Also works well, once I do some Firefox-specific magic to allow CORS or something.

Conclusion and Next Steps

So yeah, this all is pretty cool. I’ve been watching a lot of videos on LLMs, it’s really interesting to understand how the training and tuning phases work, all the data-scrubbing issues involved , and things like why LLMs hallucinate citations and other things. (BTW, AI owes a huge debt of gratitude to Wikipedia, Reddit, and Github!) AI is not scary, it’s just a tool. Even as a software engineer, I have no fear of being replaced by ChatGPT. It will just be my helper.

But, man! I sure wish GPUs with gobs of VRAM were cheaper! (Join the club, huh?)

Next Steps:

  1. Keep using and learning Ellama
  2. Investigate ollama-rs
  3. Investigate encrypted (i.e. https) connections to Ollama, unencrypted freaks me out even on my home network.
    • Investigate auth, so I can use my LLM server from elsewhere for free
  4. Tool use/MCPs with Ollama server
  5. Multimodal (e.g. images)

Upgrading to Fedora 33: Removing Your Old Swap File on EFI Machine

Fedora 33 adds a compressed-memory-based swap device using zram. Cool! Now you can remove your old swap device, if you were a curmudgeon like me and even had one in the first place.

If you are NOT on an EFI system or not using LVM, be aware of this and make changes to these steps as needed. (Specifically, the path given in step 6 will be different.)

  1. After upgrading to Fedora 33, run free. Notice that swap size is the sum of the 4G zram device plus your previous disk-based swap device. Try zramctl and lsblk commands for more info.
  2. Stop swapping to the swap device we’re about to remove. If using LVM, expect the VG and LV names to be different.
    swapoff /dev/vg0/swap
  3. If LVM, remove the no-longer-needed logical volume.
    lvremove /dev/vg0/swap
  4. Edit /etc/fstab and remove (or comment out) the line for your swap device.
  5. Edit /etc/default/grub.
    In the GRUB_CMDLINE_LINUX line, remove the “resume=” part referring to the now-gone swap partition, and the "rd.lvm.lv=” part that also refers to it.
  6. Apply above changes to actual GRUB configuration:
    grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Reboot and your system should come back up. Enjoy using that reclaimed disk space for more useful things — it’s now unused space in the LVM volume group. If you want to actually use it, look into lvextend, and also resize2fs or xfs_growfs.

Stratis 1.0 released!

We just tagged Stratis 1.0.

I can’t believe I haven’t blogged about Stratis before, although I’ve written in other places about it. We’ve been working on it for two years.

Basically, it’s a fancy manager of device-mapper and XFS configuration, to provide a similar experience as ZFS and Btrfs, but completely different under the hood.

Four things that took the most development time (so far)
  1. Writing the design doc. Early on, much of the work was convincing people the approach we wanted was a good one. We spent a lot of time discussing details among ourselves and winning over internal stakeholders (or not), but most of all, showing that we had given serious thought to various alternatives, and had spent some time to comprehend the consequences of initial design choices. Having the design doc made these discussions easier, and solicited feedback that resulted in a much better design than what we started with.
  2. Implementing on-disk metadata formats and algorithms to protect maximally against corruption and over-write. People said it would take more time than we thought and they…weren’t wrong! I still think implementing this was the right call, however.
  3. The hordes of range lists Stratis manages internally. It was probably inevitable that using multiple device-mapper layers involves a lot of range mapping. Stratis does a lot of it now, and it will be doing way more in the future, once we start using DM devices like integrity, raid, and compression. Rust really came through for us here I think. Rust’s functional aspects work very well for things like mapping and allocating.
  4. The D-Bus interface was a big effort in the pre-0.5 timeframe, but now that it is up and running it’s easy to maintain and update. We owe much of this to the quality of the dbus-rs library, and the receptivity of its author, diwic, to help us understand how to use it, and also helping to add small bits that aided our usage of D-Bus.
People to thank

Thanks to Igor Gnatenko and Josh Stone, two people who played a large part in making Rust on Fedora a reality. When I started writing the prototype for Stratis, this was a big question mark! I just hoped that the value of Rust would ensure that sooner or later Rust would be supported on Fedora and RHEL, and thanks to these two (and others, and, oh, you know, Firefox needing it…) it worked out.

I’d also like to thank the Rust community, for making such a compelling, productive systems language through friendliness and respect, sweating the details, and sharing! Like I alluded to before, Rust’s functional style was a good match for our problem space, and Rust’s intense focus on error handling also was perfect for a critical piece of software like stratisd, where what to do about errors is the most important part of what it does.

Finally, I’d like to thank the other members of the Stratis core team: Todd, Mulhern, and Tony. Stratis 1.0 is immeasurably better because of the different backgrounds and strengths we each brought to bear on developing this new piece of software. Thanks, everybody. You made 1.0 happen.

The Future

The 1.0 release marks the end of the beginning, so to speak. We just left the Shire, Frodo! Stratis is a viable product, but there’s so much more to do. Integrating more high-value device-mapper layers, more integration with other storage APIs (both “above” and “below”), more flexibility around adding and removing storage devices, while keeping the UI clean and the admin work low, is the challenge.

Stratis is going to need some major help to get there. For people interested in doing development, testing, packaging, or using Stratis, I invite you to visit our website and GitHub, or just keep tabs by following the project on Google Plus or Twitter.

Why Rust for Low-level Linux programming?

I think Rust is extremely well-suited for low level Linux systems userspace programming — daemons, services, command-line tools, that sort of thing.

Low-level userspace code on Linux is almost universally written in C — until one gets to a certain point where it’s acceptable for Python to be used. Undoubtedly this springs from Linux’s GNU & Unix heritage, but there are also many recent and Linux-specific pieces that are written in C. I think Rust is a better choice for new projects, and here’s why.

Coding is challenging because of mental context-keeping

Coding is hard and distractions are bad because of how much context the developer needs to keep straight as they look at the code. Buffers allocated, locks taken, local variables — these all create little mental things that I need to remember if I’m going to understand a chunk of code, and fix or improve it. Why have proper indentation? Because it helps us keep things straight in our heads. Why keep functions short? Same reason.

Rust reduces the amount of state I need to keep track of in my brain. It checks things that before I depended on myself to check. It gives me tools to express what I want in fewer lines, but still allows maximum control when needed. The same functionality with less code, and checks to ensure it’s better code, these make me more productive and introduce fewer bugs.

Strong types help the compiler help you

Strong typing gives the compiler information it can use to spot errors. This is important as a program grows from a toy into a useful thing. Assumptions within the code change and strong typing check the assumptions so that each version of the program globally either uses the old, or the new assumptions, but not both.

The key to this is being able to describe to the compiler the intended constraints of our code as clearly as possible.

Expressive types prevent needing type “escape hatches”

One problem with weakly-typed languages is when you need to do something that the type system doesn’t quite let you describe. This leads to needing to use the language’s escape hatches, the “do what I mean!” outs, like casting, that let you do what you need to do, but also inherently create places where the compiler can’t help check things.

Or, there may be ambiguity because a type serves two purposes, depending on the context. One example would be returning a pointer. Is NULL a “good” return value, or is it an error? The programmer needs to know based upon external information (docs), and the compiler can’t help by checking the return value is used properly. Or, say a function returns int. Usually a negative value is an error, but not always. And, negative values are nonzero so they evaluate as true in a conditional statement! It’s just…loose.

Rust distinguishes between valid and error results much more explicitly. It has a richer type system with sum types like Option and Result. These eliminate using a single value for both error and success return cases, and let the compiler help the programmer get it right. A richer type system lets us avoid needing escapes.

Memory Safety, Lifetimes, and the Borrow Checker

For me, this is another case where Rust is enabling the verification of something that C programmers learned painfully how to do right — or else. In C I’ve had functions that “borrowed” a pointer versus ones that “took ownership”, but this was not enforced in the language, only documented in the occasional comment above the function that it had one behavior or the other. So for me it was like “duh”, yeah, we notate this, have terms that express what’s happening, and the compiler can check it. Having to use ref-counting or garbage collection is great but for most cases it’s not strictly needed. And if we do need it, it’s available.

Cargo and Libraries

Cargo makes using libraries easy. Easy libraries mean your program can focus more on doing its thing, and in turn make it easier for others to use what you provide. Efficient use of libraries reduce duplicated work and keep lines of code down.

Functional Code, Functional thinking

I like iterators and methods like map, filter, zip, and chain because they make it easier to break down what I’m doing to a sequence into easier to understand fundamental steps, and also make it easier for other coders to understand the code’s intent.

Rewrite everything?

It’s starting to be a cliche. Let’s rewrite everything in Rust! OpenSSL, Tor, the kernel, the browser. Wheeee! Of course this isn’t realistic, but why do people exposed to Rust keep thinking things would be better off in Rust?

I think for two reasons, each from a different group. First, from coders coming from Python, Ruby, and JavaScript, Rust offers some of the higher-level conveniences and productivity they expect. They’re familiar with Rust development model and its Cargo-based, GitHub-powered ecosystem. Types and the borrow checker are a learning curve for them, but the result is blazingly fast code, and the ability to do systems-level things, like calling ioctls, in which a C extension would’ve been called for — but these people don’t want to learn C. These people might call for a rewrite in Rust because it brings that component into the realm of things they can hack on.

Second, there are people like me, people working in C and Python on Linux systems-level stuff — the “plumbing”, who are frustrated with low productivity. C and Python have diametrically-opposed advantages and disadvantages. C is fast to run but slow to write, and hard to write securely. Python is more productive but too slow and RAM-hungry for something running all the time, on every system. We must deal with getting C components to talk to Python components all the time, and it isn’t fun. Rust is the first language that gives a system programmer performance and productivity. These people might see Rust as a chance to increase security, to increase their own productivity, to never have to touch libtool/autoconf ever again, and to solve the C/Python dilemma with a one language solution.

Incremental Evolution of the Linux Platform

Fun to think about for a coder, but then, what, now we have C, Python, AND Rust that need to interact? “If only everything were Rust… and it would be so easy… how hard could it be?” 🙂 Even in Rust, a huge, not-terribly-fun task. I think Rust has great promise, but success lies in incremental evolution of the Linux platform. We’ve seen service consolidation in systemd and the idea of Linux as a platform distinct from Unix. We have a very useful language-agnostic IPC mechanism — DBus — that gives us more freedom to link things written in new languages. I’m hopeful Rust can find places it can be useful as Linux gains new capabilities, and then perhaps converting existing components may happen as maintainers gain exposure and experience with Rust, and recognize its virtues.

The Future

Rust is not standing still. Recent developments like native debugging support in GDB, and the ongoing MIR work, show that Rust will become even better over time. But don’t wait. Rust can be used to rapidly develop high-quality programs today. Learning Rust can benefit you now, and also yield dividends as Rust and its ecosystem continue to improve.

I’m Part of SFConservancy’s GPL Compliance Project for Linux

I believe GPL enforcement in general, and specifically around the Linux kernel, is a good thing. Because of this, I am one of the Linux copyright holders who has signed an agreement for the Software Freedom Conservancy to enforce the GPL on my behalf. I’m also a financial supporter of Conservancy.

That I hold any copyright at all to the work I’ve done is actually somewhat surprising. Usually one of the documents you sign when beginning employment at a company is agreeing that the company has ownership of work you do for them, and even things you create off-hours. Fair enough. But, my current employer does not do this. Its agreement lets its employees retain individual copyright to their work for open source development projects. I haven’t checked, but I suspect the agreement I signed with my less-steeped-in-F/OSS previous employers did not.

It’s something to ask about when considering accepting a new position.

I consider myself an idealist, but not a zealot. For projects I’ve started, I’ve used MIT, AGPLv3, MPLv2, GPLv2, GPLv3, LGPLv2, and Apache 2.0 licenses, based on the individual circumstances.

I believe if you’re going to build upon someone else’s work, it’s only fair to honor the rules they have set — the license — that allow it. Everyone isn’t doing that now with Linux, and are using the ambiguity over what is a “derived work” as a fig leaf. Enforcing the GPL is the only way to ensure the intent of the license is honored. We’ll also hopefully eventually gain some clarity on exactly what constitutes a derived work from the courts.

Please consider becoming a Conservancy Supporter.

iSNS support coming soon for LIO in Fedora

target-isns recently was added to Rawhide, and will be in a future Fedora release. This add-on to LIO allows it to register with an iSNS server, which potential initiators can then query for available targets. (On Fedora, see isns-utils for both the server, and client query tools.) This removes one of the few remaining areas that other target implementations have been ahead of LIO.

Kudos and thanks to Christophe Vu-Brugier for writing this useful program!

Some targetcli and TCMU questions

Just got an email full of interesting questions, I hope the author will be ok with me answering them here so future searches will see them:

I searched on internet and I don’t find some relevant info about gluster api support via tcmu-runner. Can you tell me please if this support will be added to the stable redhat targetcli in the near future? And I want to know also which targetcli is recommended for setup (targetcli or targetcli-fb) and what is the status for targetcli-3.0.

tcmu-runner is a userspace daemon add-on to LIO that allows requests for a device to be handled by a user process. tcmu-runner has early support for using glfs (via gfapi). Both tcmu-runner and its glfs plugin are beta-quality and will need further work before they are ready for stable Fedora, much less a RHEL release. tcmu-runner just landed in Rawhide, but this is really just to make it easier to test.

RHEL & Fedora use targetcli-fb, which is a fork of targetcli, and what I work on. Since I’m working on both tcmu-runner and targetcli-fb, targetcli-fb will see TCMU support very early.

The -fb packages I maintain switched to a “fbXX” version scheme, so I think you must be referring to the other one 🙂 I don’t have any info about the RTS/Datera targetcli’s status, other than nobody likes having two versions, the targetcli maintainer and I have discussed unifying them into a common version, but the un-fun work of merging them has not happened yet.

RHEL 7.2 has an updated kernel target

As mentioned in the beta release notes, the kernel in RHEL 7.2 contains a rebased LIO kernel target, to the equivalent of the Linux 4.0.stable series.

This is a big update. LIO has improved greatly since 3.10. It has added support for SCSI features that enable VMWare VAAI support, as well as data integrity (DIF), and significant iSER work, for those of you using Infiniband. (SRP is also supported, as well as iSCSI and FCoE, of course.)

Note that we still do not ship support for the Fibre Channel qla2xxx fabric. It still seems to be something storage vendors and integrators want, more than a feature our customers are telling us they want in RHEL.

(On a side note, Infiniband hardware is pretty affordable these days! For all you datacenter hobbyists who have a rack in the garage, I might suggest a cheap previous-gen IB setup and either SRP or iSER as the way to go and still get really high IOPs.)

Users of RHEL 7’s SCSI target should find RHEL 7.2 to be a very nice upgrade. Please try the beta out and report any issues you find of course, but it’s looking really good so far.