...making Linux just a little more fun!
[ In reference to "Measuring TCP Congestion Windows" in LG#136 ]
René Pfeiffer [lynx at luchs.at]
[[[ Greetings to Indonesia! Thanks to this thread, I've learned the word "Balasan", which seems to mean "re:". Is this a word in Malay, or Bahasa Indonesia? -- Kat ]]]
Hello!
On Feb 15, 2008 at 1135 +0700, Niko Wilfritz Sianipar Sianipar appeared and said:
> I,ve read your article about measuring tcp congestion window. Maybe I > can ask you, how to use tcp_info structure in my Apache code (more > specifically in sendfile_it_all function in core_filters.c. I want to > get the time to transfer last byte sent in order to schedule the > packet sent to client according to that metric), cause I really realy > confused with the Apache source code. Or, do you know what part in > Apache that provide the time to transfer last by sent to client.
The tcp_info structure contains statistics derived from an active TCP connection. getsockopt() fills the structure with data. An example function call from my article in #136 looks like this:
getsockopt( tcp_work_socket, SOL_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length );
I looked into the code for core_filters.c in a httpd 2.2.8 distribution. Basically the sendfile_it_all() function reads a file and sends it to the network by using sendfile(). There's not much you can tune there since sendfile() does most of the buffering. You could periodically call getsockopt() with the TCP_INFO option, but you'd only get the TCP parameters. You cannot influence the TCP connection when it transfers date, it is handled by the kernel. You can only use one of the TCP congestions algorithms and set some parameters of the TCP stack (either through /proc/sys/net/... or by using socket options such as TCP_CORK).
If you want to do any timing you have to do the buffering yourself, send chunks of data and take the time after every buffer transmit. That's what I did in my example code, but that's not what you want to do in production code. The Linux kernel has the TCP stack and should deal with the details. The kernel also keeps track of TCP parameters used with recent clients. An application usually doesn't see all of this.
As far as the Apache code is concerned, I don't know where or whether Apache stores the time of transfers. There are some options for the socket code, so I'd look for the persistant connections (KeepAlive, etc.) or all things that deal with the socket handling.
Best, René.
René Pfeiffer [lynx at luchs.at]
On Feb 19, 2008 at 2100 +0700, Niko Wilfritz Sianipar Sianipar appeared and said:
> How to use getsockopt in core_filters.c (in sendfile_it_all() > function)? Because we can't use c->client_socket->sockdes in > getsockopt. it will error at compiling the httpd.
Can you show me the source of this file along with your modifications? What does the compiler complain about?
> Actually, I'm doing a task about Apache that set socket priority > according to the throughput of socket's connection (not need to get > the exact value just need an approach). And I need the tcp_info > structure to find out the throughput.
You can measure the throughput yourself if you send out chunks of a certain size and measure the time. The problem is that the kernel will queue packets and that your measurement is a rough estimate.
I have no clear idea what you mean by setting the socket priority. Do you have something like this http://curl.haxx.se/mail/lib-2005-03/0024.html in mind?
> The idea you give to me is using the getsockopt, and I will do it > before the sendfile() function. And just before the sendfile() call, I > also call the setsockopt() to set the socket priority with SO_PRIORITY > option (Am I right???), then sendfile() will be executed.
If you call getsockopt() before sendfile() you will only get a snapshot of the TCP parameters before the file transmission. Once you call sendfile() userspace is out of business and I guess you can't peek into the transmission anymore. I'd say that you have to do the buffering yourself if you want to keep track of the TCP parameters.
> I use tc to make the PRIO qdisc. > That's all what I'm gonna do. And I hope you can also help me with this explanation.
Well, I'm trying, but so far I have only a vague idea of what you wish
to code. 
Best, René.
René Pfeiffer [lynx at luchs.at]
On Feb 21, 2008 at 1211 +0700, Niko Wilfritz Sianipar Sianipar appeared and said:
> What buffering you mean here? I still confused with the buffering term > here. Can you write some example code or something.
The example code is in the articles I wrote. If you look up the man page of sendfile() it says:
sendfile() copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read(2) and write(2), which would require transferring data to and from user space.
This means that as soon as you call sendfile() your file goes from the filesystem to the socket and your code won't get any notice about how many bytes are transferred within a given time frame. This in turn means that you have to use read(), write() and your own buffer if you want to do any measurement such as "bytes per second".
My advise is to let the kernel handle this if you are going to use sendfile() anyway. The kernel knows best what to do during the TCP transmission. If you want to play with TCP parameters during the transmission you're better of writing your own TCP/IP stack or coding a new TCP congestions algorithm kernel module. Especially the latter is easier since you are in kernel space and can easily query the kernel's internal structures without resorting to getsockopt() and TCP_INFO.
> About the file that I've modified I will write as soon as possible here. > Squid configurationr your patience..Still help me with this task please... > Wait for a minute..
Waiting is the easy part.
Best, René.
René Pfeiffer [lynx at luchs.at]
On Feb 21, 2008 at 1802 +0700, Niko Wilfritz Sianipar Sianipar appeared and said:
> This I attach the file that I've modified in core_filters.c. Still to > much bug there, and I hope you can help me with your experience.
Yes, you have to be careful about the variables you introduce.
> Please look at sendfile_it_all() function, also in the "include file" part. > I also confuse with the #include part. The error maybe come from there too. > Here, I will try to modifying it too with all my "POWER". > Thanks.
I looked at your code. As far as I understand you do the following in sendfile_it_all():
- Enter an endless loop - Get the TCP parameters by requesting a TCP INFO structure with getsockopt() - Compute the throughput - Set the socket priority according to throughput classes - Call apr_socket_sendfile() to send the file to the socket - Return if apr_socket_sendfile() sent all bytes of the file or returned APR_SUCCESS
I doubt that your throughput measurements will yield useful data. In addition to that the SO_PRIORITY may influence the ToS field for outgoing packets, but you could do this by using Netfilter. It's still only an advice to all network equipment outside your server box, and I don't believe that you will gain performance by doing this.
Best, René.
core_filters.c core_filters_modified.c
Rick Moen [rick at linuxmafia.com]
Quoting Ren? Pfeiffer (lynx@luchs.at):
> /* Copyright 2001-2005 The Apache Software Foundation or its licensors, as > * applicable. > * > * Licensed under the Apache License, Version 2.0 (the "License"); > * you may not use this file except in compliance with the License. > * You may obtain a copy of the License at > * > * http://www.apache.org/licenses/LICENSE-2.0 > * > * Unless required by applicable law or agreed to in writing, software > * distributed under the License is distributed on an "AS IS" BASIS, > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > * See the License for the specific language governing permissions and > * limitations under the License. > */
Rene, I've manually approved that post, even though it was 2x over the size limit for this mailing list. It would be appreciated, in similar situations, if you would please post URLs from which such files can be fetched, rather than file-attaching them inline. Thanks.
[[[ I've clipped that message and turned the attached text into clickable links. -- Kat ]]]
René Pfeiffer [lynx at luchs.at]
On Feb 26, 2008 at 1144 +0700, Niko Wilfritz Sianipar Sianipar appeared and said:
> Can I use tcp_info.tcpi_snd_cwnd/tcp_info.tcpi_rtt to get throughput in Bytes/second?
AFAIK you can't. The ratio between the sender's congestion window and the round trip time per packet is not the throughput.
> And how to use read() and write() function to calculate the time of > transfer of a chunk data?
I used both functions to fill/send a buffer and took the time before and after calling the functions. This isn't the throughput either, because the kernel keeps buffers for every socket. You can only estimate the throughput.
> Have you receive my attached file??
Yes, and I already commented it.
Best, René.