Visitors To This Post

Search This Blog

Showing posts with label Ultra Low Latency. Show all posts
Showing posts with label Ultra Low Latency. Show all posts

Saturday, February 1, 2020

Ultra Low Latency Server Huge-page Config


Ultra Low Latency Server Huge-page Config






#

# Uncomment the following to stop low-level messages on console
kernel.printk = 3 4 1 3

# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
net.ipv4.tcp_fin_timeout = 1
#

# Increase maximum amount of memory allocated to shm
# Increase size of file handles and inode cache
fs.file-max = 209715200000
#

# Increase size of Kernel Memory Sharing
kernel.shmmax = 51539607552
kernel.shmmni = 515396075
kernel.shmall = 51539607552

vm.hugepages_treat_as_movable=24576
vm.nr_hugepages=24576
vm.nr_hugepages_mempolicy=24576
vm.nr_overcommit_hugepages=24576
vm.hugetlb_shm_group=0

#
# This will increase the amount of memory available for socket input/output queues
# Increase number of incoming connections
net.core.somaxconn = 65535
net.ipv4.tcp_rmem = 4096 51539607552 51539607552

# Maximum Socket Receive Buffer
net.core.rmem_max = 51539607552

# Default Socket Receive Buffer
net.core.rmem_default = 51539607552
net.ipv4.tcp_wmem = 4096 65536 51539607552

# Maximum Socket Send Buffer
net.core.wmem_max = 51539607552

# Default Socket Send Buffer
net.core.wmem_default = 31457280
net.core.optmem_max = 51539607552

net.ipv4.tcp_max_orphans = 51539607552
net.ipv4.tcp_max_syn_backlog = 51539607552

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 51539607552

##############################################################3

# Do less swapping
vm.swappiness = 0
vm.dirty_ratio = 1
vm.dirty_background_ratio = 1

### GENERAL NETWORK SECURITY OPTIONS ###

# Allowed local port range
net.ipv4.ip_local_port_range = 1 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

##############################################################3

### TUNING NETWORK PERFORMANCE ###

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPV4/IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1

###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
net.ipv4.conf.all.log_martians = 1
#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all
# Debian kernels have this set to 0 (disable the key)
# See https://www.kernel.org/doc/Documentation/sysrq.txt
# for what other values do
kernel.sysrq=0

###################################################################
# Protected links
#
# Protects against creating or following links under certain conditions
# Debian kernels have both set to 1 (restricted)
# See https://www.kernel.org/doc/Documentation/sysctl/fs.txt
fs.protected_hardlinks=0
fs.protected_symlinks=0
#
###################################################################

Optimizing and Tuning R815-Server via Ultra/Low-Latency Configurations:

0. tuned-adm profile latency-performance 
0.1 for MF in `find /proc/irq -name *smp_affinity` ; do awk -F, \ '{for(i=1;i<NF;i++)printf("00000000,");printf("%8.8x\n",and(0x00000001, strtonum("0x"$NF)))}' \ $MF > $MF ; done

1. find /sys/kernel/slab -name 'cpu_partial' -exec sh -c 'echo 0 > {}' \;
2. cset shield --cpu 1-63 --kthread=on 
3.0 cset proc --move --pid=$$ --threads --toset=root
3.1 cset proc --move --pid=$$ --threads --toset=user
. grub2-mkconfig -o /boot/grub2/grub.cfg
or
.grub-mkconfig -o /boot/grub/grub.cfg




dnf install -y libhugetlbfs-utils


This is the 'Magic file'
/etc/sysctl.conf


The goal is to make the output of this program "happy"
sudo hugeadm --set-recommended-shmmax


This sets changes in motion immediately, though a reboot is suggested once everything is configured.
sysctl -p


We will make changes by echoing options to sysctl.conf
echo "vm.hugetlb_shm_group = 0" >> /etc/sysctl.conf 
echo "vm.min_free_kbytes = 112640" >> /etc/sysctl.conf 
echo "vm.nr_hugepages = 8600" >> /etc/sysctl.conf 
echo "kernel.shmmax = 18035507200" >> /etc/sysctl.conf 
echo "vm.swappiness = 0" >> /etc/sysctl.conf