{"id":1802,"date":"2026-01-22T19:53:25","date_gmt":"2026-01-22T19:53:25","guid":{"rendered":"https:\/\/editjournal.redakt.eu\/faxmodem\/?p=1802"},"modified":"2026-01-22T19:53:25","modified_gmt":"2026-01-22T19:53:25","slug":"debug-freebsd-unresponsive-io","status":"publish","type":"post","link":"https:\/\/editjournal.redakt.eu\/faxmodem\/blog\/servers\/freebsd\/debug-freebsd-unresponsive-io\/","title":{"rendered":"FreeBSD gets unresponsive on commands touching disk"},"content":{"rendered":"<p>Imagine your server hangs even on simple, seemingly unrelated commands such as <code>cat file<\/code>, <code>ps aux<\/code> or <code>top<\/code>. This is the issue I've been dealing with today. Short version: it was MySQL\/MariaDB server clogging the VPS I\/O.<\/p>\n<p><strong>Important<\/strong>: even though a terminal window with an established SSH connection would freeze, I could always open a new Terminal window with a new SSH connection to try a different command, or to debug the one stalled in the first windown.<\/p>\n<p>Below are steps to debug.<\/p>\n<h2>Confirm this actually is I\/O \u2013 iostat<\/h2>\n<p>Run <code>iostat -x 1<\/code> and check <code>%b<\/code>.<\/p>\n<pre><code>root@:~ # iostat -x 1\n                        extended device statistics  \ndevice       r\/s     w\/s     kr\/s     kw\/s  ms\/r  ms\/w  ms\/o  ms\/t qlen  %b  \nda0          201      98   7201.8   5448.0     5    30     0    13    1  98 \ncd0            0       0      0.0      0.0     0     0     0     0    0   0 \npass0          0       0      0.0      0.0     0     0     0     0    0   0 \npass1          0       0      0.0      0.0     0     0     0     0    0   0 \n                        extended device statistics  \ndevice       r\/s     w\/s     kr\/s     kw\/s  ms\/r  ms\/w  ms\/o  ms\/t qlen  %b  \nda0          199     101   7276.7   6159.6     4    11     0     6    0  97 \ncd0            0       0      0.0      0.0     0     0     0     0    0   0 \npass0          0       0      0.0      0.0     0     0     0     0    0   0 \npass1          0       0      0.0      0.0     0     0     0     0    0   0 \n                        extended device statistics  \ndevice       r\/s     w\/s     kr\/s     kw\/s  ms\/r  ms\/w  ms\/o  ms\/t qlen  %b  \nda0          195     105   7839.5   6175.6     4    22     0    10    0  97 \ncd0            0       0      0.0      0.0     0     0     0     0    0   0 \npass0          0       0      0.0      0.0     0     0     0     0    0   0 \npass1          0       0      0.0      0.0     0     0     0     0    0   0 \n                        extended device statistics  \ndevice       r\/s     w\/s     kr\/s     kw\/s  ms\/r  ms\/w  ms\/o  ms\/t qlen  %b  \nda0          166     125   6351.7   6129.4     5    53     0    25    0  99 \ncd0            0       0      0.0      0.0     0     0     0     0    0   0 \npass0          0       0      0.0      0.0     0     0     0     0    0   0 \npass1          0       0      0.0      0.0     0     0     0     0    0   0 \n^C<\/code><\/pre>\n<p>(Output truncated, but you get the idea.)<\/p>\n<p>The above does show very heavy disk utilization, and write latency spikes:<\/p>\n<ul>\n<li><code>%b<\/code> pushing 100% almost constantly confirms the disk is busy almost all the time<\/li>\n<li><code>ms\/w<\/code> jumping to 53 ms shows the writes are slow<\/li>\n<li>if <code>qlen<\/code> goes up occasionally, that means queued I\/O bursts (backpressure)<\/li>\n<\/ul>\n<h2>Confirm this actually is I\/O \u2013 vmstat<\/h2>\n<p>Run <code>vmstat 1<\/code> and check column for <code>da0<\/code> from the previous test.<\/p>\n<pre><code>root@:~ # vmstat 1\n procs    memory    page                      disks       faults       cpu\n r  b  w  avm  fre  flt  re  pi  po   fr   sr  da0  cd0   in   sy   cs us sy id\n 0  0  0 1.1G 455M    0   0   0   0 1.9k   60  300    0  274  654 2.1k  1  1 96\n 0  0  0 1.1G 457M    0   0   0   0 2.6k   60  299    0  288  537 1.8k  0  5 93\n 1  0  0 1.1G 457M    0   0   0   0 2.9k   60  301    0  298  467 1.9k  0  0 99\n 0  0  0 1.1G 456M    0   0   0   0 2.0k   60  298    0  294  413 1.6k  0  5 94\n^C<\/code><\/pre>\n<p>(Output truncated, but you get the idea.)<\/p>\n<p>As you can see here, the same <code>da0<\/code> is busy at around ~300.<\/p>\n<h2>Do you run a database on this server?<\/h2>\n<p>On FreeBSD, MariaDB is almost always configured with durability-heavy defaults.<\/p>\n<p>Check the status: <code>service mysql-server onestatus<\/code><\/p>\n<pre><code>root@:~ # service mysql-server onestatus\nmysql is running as pid 2313.<\/code><\/pre>\n<p>Run <code>procstat kstack &lt;PID&gt; | egrep &#039;bwait|bufwait&#039;<\/code> to spot I\/O buffer waits. Grep is used as a fast filter. Use a for loop to check multiple snapshots.<\/p>\n<pre><code>for i in 0 1 2 3 4 5 6 7 8 9; do procstat kstack &lt;PID&gt; | egrep &#039;bwait|bufwait&#039;; sleep 2; done<\/code><\/pre>\n<p>If this prints anything, at least one thread is blocking filesystem and\/or I\/O. Example:<\/p>\n<pre><code>root@:~ # procstat kstack 2313 | egrep &#039;bwait|bufwait&#039;\n 2313 100447 mariadbd            -                   mi_switch _sleep bwait bufwait cluster_read ffs_read VOP_READ_APV vn_read vn_io_fault_doio vn_io_fault1 vn_io_fault dofileread kern_preadv sys_pread amd64_syscall fast_syscall_common <\/code><\/pre>\n<h2>So this is MariaDB?<\/h2>\n<p>Check if your MariaDB server is tuned appropriately.<\/p>\n<ol>\n<li>\n<p>How much RAM does this server have?<\/p>\n<pre><code>root@:~ # sysctl hw.physmem\nhw.physmem: 2111275008<\/code><\/pre>\n<p>2 GB<\/p>\n<\/li>\n<li>\n<p>How large is the database?<\/p>\n<pre><code>root@:~ # du -sh \/var\/db\/mysql\n40G \/var\/db\/mysql<\/code><\/pre>\n<\/li>\n<li>\n<p>Check <code>innodb_buffer_pool_size<\/code>, <code>innodb_flush_log_at_trx_commit<\/code> and <code>sync_binlog<\/code>.<\/p>\n<pre><code>root@:~ # mysql -e \"SELECT @@innodb_buffer_pool_size\/1024\/1024 AS buffer_pool_mb; SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit'; SHOW VARIABLES LIKE 'sync_binlog';\"\n\n+----------------+\n| buffer_pool_mb |\n+----------------+\n|   128.00000000 |\n+----------------+\n+--------------------------------+-------+\n| Variable_name                  | Value |\n+--------------------------------+-------+\n| innodb_flush_log_at_trx_commit | 1     |\n+--------------------------------+-------+\n+---------------+-------+\n| Variable_name | Value |\n+---------------+-------+\n| sync_binlog   | 0     |\n+---------------+-------+<\/code><\/pre>\n<p>The above shows:<\/p>\n<ul>\n<li>InnoDB buffer pool is 128 MB which is tiny<\/li>\n<li><code>innodb_flush_log_at_trx_commit<\/code> = 1 means doing fsync on every commit<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>So we\u2019re running a 40 GB dataset with effectively 128 MB of cache. That guarantees constant cache misses. On top of that, trx_commit=1 forces frequent log flushes, which is a problem on storage with poor sync latency.<\/p>\n<h2>Fix<\/h2>\n<ol>\n<li>Increase innodb_buffer_pool_size<br \/>\nOn a 2 GB box, a sane starting point is 1G if this machine is primarily for MySQL\/MariaDB.<\/li>\n<li>Relax flush policy (optional but very impactful on a VPS)<br \/>\nSet innodb_flush_log_at_trx_commit=2 to avoid fsync on every commit.<br \/>\nTradeoff: you can lose up to ~1 second of transactions if the host crashes.<\/li>\n<\/ol>\n<p>Add this to <code>\/usr\/local\/etc\/mysql\/conf.d\/server.cnf<\/code> under <code>[mysqld]<\/code><\/p>\n<pre><code># Cache more of the working set in RAM (big read I\/O reduction)\ninnodb_buffer_pool_size = 1G\n\n# Reduce fsync pressure (reduce stalling on VPS storage)\ninnodb_flush_log_at_trx_commit = 2<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Imagine your server hangs even on simple, seemingly unrelated commands such as cat file, ps aux or top. This is the issue I&#8217;ve been dealing with today. Short version: it was MySQL\/MariaDB server clogging the VPS I\/O. Important: even though a terminal window with an established SSH connection would freeze, I could always open a&hellip; <a class=\"more-link\" href=\"https:\/\/editjournal.redakt.eu\/faxmodem\/blog\/servers\/freebsd\/debug-freebsd-unresponsive-io\/\">Continue reading <span class=\"screen-reader-text\">FreeBSD gets unresponsive on commands touching disk<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1258],"tags":[932,817],"class_list":["post-1802","post","type-post","status-publish","format-standard","hentry","category-freebsd","tag-mariadb","tag-vps","entry"],"_links":{"self":[{"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/posts\/1802","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/comments?post=1802"}],"version-history":[{"count":2,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/posts\/1802\/revisions"}],"predecessor-version":[{"id":1804,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/posts\/1802\/revisions\/1804"}],"wp:attachment":[{"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/media?parent=1802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/categories?post=1802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/editjournal.redakt.eu\/faxmodem\/wp-json\/wp\/v2\/tags?post=1802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}