FlightAware operates many FreeBSD severs and includes some useful tuning tips for other FreeBSD operators that need a high capacity configuration. Most operating system and server settings are extremely low and will cause failures if they're not raised substantially. Unfortunately, it is often difficult to find these configuration limitations until you experience a failure.
FlightAware runs the following configurations on machines ranging from 2x2GHz/64GB to 32x3GHz/384GB, so they can be dropped in to most configurations without a problem.
This page was last updated October 2013 and reflects our usage of FreeBSD 8.2.
/boot/loader.conf
Don't remove your ZFS settings that might be at the top of this file.
The kern.ipc.maxsockets and kern.ipc.maxsockbuf settings are for any large network server.
accf_http is a socket filter that buffers incoming connctions until a complete HTTP request arrives. The accf_http_load setting loads the module and prevents an Apache warning on start. FlightAware does not expose Apache to the Internet; it's proxied by Varnish. Accordingly, this setting is unnecessary for us but useful for others.
The rest of the semaphore/shared memory/swap settings are to allow PostgreSQL to have large (we use 16-72GB) shared memory segments. The postgres web site has documentation on kernel resource settings..
kern.maxusers="1024" kern.ipc.semmns="2048" kern.ipc.semmni="128" kern.ipc.shmall="33554432" kern.ipc.shmseg="1024" kern.ipc.shmmax=137438953472 kern.ipc.maxsockets="256000" kern.ipc.maxsockbuf="2621440" kern.maxswzone="335544320" accfhttp_load="YES" hw.igb.enable_msix=0
/etc/sysctl.conf
The shm settings are for PostgreSQL (see above).
vm.pmap.shpgperproc settings are only required for FreeBSD 8 and below.
# set max shared memory segment size to 512 megs, 512 * 1024 * 1024, we need a # lot for PostgreSQL kern.ipc.somaxconn=32768 kern.maxfiles=400000 kern.ipc.shmmax=68719476736 # set the maximum number of pages of shared memory, # this should be at least # (max mem you could possible allocate / page size in bytes), i.e. # at least shmmax / pagesize # kern.ipc.shmall=16777216 kern.ipc.semmap=256 # # lock shared memory from being swapped out # kern.ipc.shm_use_phys=1 # http://spatula.net:8000/blog/2007/04/freebsd-network-performance-tuning.html net.inet.tcp.recvspace=78840 net.inet.tcp.sendspace=78840 net.local.stream.sendspace=82320 net.local.stream.recvspace=82320 net.inet.tcp.local_slowstart_flightsize=10 kern.maxfilesperproc=104856 kern.threads.max_threads_per_proc=16384 kern.ipc.nmbclusters=66560 vm.pmap.shpgperproc=500 net.inet.icmp.icmplim=1000
