TCPDump Capture HTTP GET/POST requests

原文链接：https://www.middlewareinventory.com/blog/tcpdump-capture-http-get-post-requests-apache-weblogic-websphere/

概述

TCPDUMP is a swiss army knife for all the administrators and developers when it comes to troubleshooting. This post is written for the people who work in middleware technologies. Web servers such as Apache, NGINX, Oracle HTTP, IHS web servers and  application servers such as Weblogic, Websphere, Tomcat, Jboss Consider yourself in any of the following scenarios

  1. You want to monitor thetraffic inflow and outflowof Apache httpd server on any specific port like port 80 or 443
  2. You have totrack the HTTP calls between web and application servers (or) to make sure that proxy is working fine.
  3. When you want to examine the presence and values of  specific headers like

X-Forwarded-For X-Forwarded-Proto X-Forwarded-Port X-Forwarded-Host X-Frame-Options X-Forwarded-By ETag

  1. When you want to review the Cookies being placed in the request and response
  2. when you want to review the data which is being posted to the server in the POST method like

Content-type: multipart/form-data;   Content-Type: application/x-www-form-urlencoded Content-Type: application/json Content-Type: text/plain

  1. To track the incoming web service call, made using SOAP UI (or) any web service client.

Table of Contents

So, How do you do that?

The answer is TCPDUMP. TCPDUMP is mostly misconceived as a network engineer’s subject and it displays some incomprehensible binary data that none of us could understand. With proper tools and little knowledge about protocols, anyone can easily make use of it and feel the magic lies within.

Be informed that the industry standards have changed for good and the HTTPS is becoming a basic requirement for all webservices and websites. So it might make your troubleshooting little hard, since the packets are encrypted.  There is a solution to decrypt HTTPS traffic Refer my another article on How to decrypt HTTPS traffic to see headers and request/response content

The Objective

In this post we are going to see how middleware administrators (or) developers could use tcpdump to accomplish their troubleshooting drama. We are going to discuss the following items , practically as much as possible.

  • How to monitor/track HTTP and HTTPS calls with tcpdump in weblogic,websphere,tomcat application servers and web servers like Apache which runs on LINUX platform
  • How to tamper and read the incoming and outgoing HTTP traffic to our applications deployed in weblogic
  • How to dig into the incoming (or) outgoing HTTP traffic and take a look at the concrete elements of HTTP protocol such asheaders, cookies, request body as they gets transmitted.
  • How to intercept the HTTP traffic initiated from the browser (or) SOAP UI to application server and sneak a peek into the content like Request Body likeXML,JSONand Username and Password etc.
  • How to monitor the incoming SOAP web service request body (or) request XML
  • More practical examples on how to use TCPDUMP to analyze the HTTP traffic

Before we proceed

Some basics about how to run tcpdump in your server in the right way. Make sure tcpdump is installed and configured properly [root@mwiws01 ~]# tcpdump –version tcpdump version 4.9.2 libpcap version 1.5.3 OpenSSL 1.0.2k-fips 26 Jan 2017 Use the right interface name (or) use any in the interface name. To Get the interface name of your IP which you need to specify it in the tcpdump command.  you can execut the command ifconfig (or) ip a In my case, My web server IP is 192.168.10.10 so I should pick and use the interface name of the same enp0s8

Once you found your interface. In the requests you are going to track. you must mention the interface name tcpdump -i enp0s8 you can optionally monitor all the available interfaces by mentioning tcpdump -i any Now we are All ready to Play.

TCPDUMP Examples

TCPDUMP does the same job irrespective to what technology (or) server you are using it for. In other words, if you would like capture HTTP calls for Apache.  You mostly going to be using the port 80 or 443. If you would like to capture the traffic of weblogic (or) Websphere or any application servers. All you have to do is change the port number in which your application server is listening Though we have given examples for both web and application server here.  All you should be aware of is that. There is nothing specific to technology. We just change the port number (or) interface. That’s all.

How to capture All incoming  HTTP GET  traffic (or) requests

tcpdump -i enp0s8 -s 0 -A ’tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420' Explanation:- tcp[((tcp[12:1] & 0xf0) » 2):4] first determines the location of the bytes we are interested in (after the TCP header) and then selects the 4 bytes we wish to match against. To know more about how this segment syntax has been derived. refer this link

Here 0x47455420 depicts the ASCII value of  characters  'G' 'E' 'T' ' '

CharacterASCII Value
G47
E45
T54
Space20

Refer this ASCII map table for more reference

How to capture All incoming HTTP POST requests

tcpdump -i enp0s8 -s 0 -A ’tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354' Here 0x504F5354 represents  the ASCII value of  'P' 'O' 'S' 'T' Sample Output [root@mwiws01 ~]# tcpdump -i enp0s8 -s 0 -A ’tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes 08:12:59.552588 IP 192.168.10.1.60651 > mwiws01.http: Flags [P.], seq 1817631852:1817632015, ack 3385979723, win 4117, options [nop,nop,TS val 399453898 ecr 6715402], length 163: HTTP: POST /new.html HTTP/1.1 E…..@.@..C.. …

…PlV.l…K……….. …..fx POST /new.html HTTP/1.1 Host: 192.168.10.10 User-Agent: curl/7.54.0 Accept: / X-Requested-By: middlewareinventory TestHeader: TestValue MyName: SaravAK

^C 1 packet captured 1 packet received by filter 0 packets dropped by kernel [root@mwiws01 ~]#

How to capture only HTTP GET requests Incoming to port 80 ( Apache/NGINX)

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420' Sample Output [root@mwiws01 ~]# tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes 06:50:12.424996 IP 192.168.10.1.58034 > mwiws01.http: Flags [P.], seq 1518079346:1518079506, ack 1444634698, win 4117, options [nop,nop,TS val 394486908 ecr 1748275], length 160: HTTP: GET /new.html HTTP/1.1 E…_.@.@.E7.. …

…PZ|.rV.`J…..u….. ..d|…3GET /new.html HTTP/1.1 Host: 192.168.10.10 User-Agent: curl/7.54.0 Accept: / X-Requested-By: middlewareinventory TestHeader: TestValue MyName: Sarav

How to capture only HTTP POST requests Incoming to port 80 ( Apache/NGINX)

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354' Sample Output [root@mwiws01 ~]# tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes 08:16:50.394502 IP 192.168.10.1.60759 > mwiws01.http: Flags [P.], seq 2281273674:2281273837, ack 1974611475, win 4117, options [nop,nop,TS val 399670036 ecr 6946244], length 163: HTTP: POST /new.html HTTP/1.1 E…G.@.@.]… …

.W.P..yJu…….x…… ..{..i..POST /new.html HTTP/1.1 Host: 192.168.10.10 User-Agent: curl/7.54.0 Accept: / X-Requested-By: middlewareinventory TestHeader: TestValue MyName: SaravAK

^C 1 packet captured 1 packet received by filter 0 packets dropped by kernel

How to capture only HTTP GET calls Incoming to port 443 ( Apache/NGINX)

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420'

How to capture only HTTP POST calls Incoming to port 443 ( Apache/NGINX)

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354'

How to capture both HTTP GET (or) POST Incoming calls to port 80 (or) 443 ( Apache/NGINX) Originating from 192.168.10.1 Host**

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 or tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354’ and host 192.168.10.1

How to capture a Complete HTTP Transmission, incoming and outgoing Including both HTTP Request and Response.  Associated with a Single Client  along with HTML page data ( GET & POST ) on port 80

Let suppose I access a page hosted in 192.168.10.10 web server from my base machine with ip address 192.168.10.1.  using both GET and POST methods.  How do we track the HTTP request and response between the server and the client

tcpdump -i enp0s8 -s 0 -A ’tcp dst port 80 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x3C21444F and host 192.168.10.1' Here additionally, we are using two more ASCII values to capture the outgoing HTML files and HTTP calls they are as follows. 0x3C21444F represents the ASCII value of  '<' 'D' 'O' 'C' this is to capture the outgoing HTML file 0x48545450 represents the ASCII value of 'H' 'T' 'T' 'P' this is to capture the outgoing HTTP traffic (HTTP response) Trying with GET

Trying with POST In the preceding illustrations,  you can see the output of tcpdump command (to the right) shows the REQUEST and RESPONSE data along with theHTML code. You can also find it aligning to the CURL’s verbose output

How to monitor all the incoming HTTP Request URL’s (POST or GET)

tcpdump -i enp0s8 -s 0 -v -n -l | egrep -i “POST /|GET /|Host:”

How to  capture HTTP Passwords in POST Requests

tcpdump -i enp0s8 -s 0 -A -n -l | egrep -i “POST /|pwd=|passwd=|password=|Host:”

How to capture the Cookies from Server and from Client ( Request & Response)

tcpdump -i enp0s8 -nn -A -s0 -l | egrep -i ‘Set-Cookie|Host:|Cookie:’

How to Filter HTTP User Agents

Extract HTTP User Agent from HTTP request header. tcpdump -vvAls0 | grep ‘User-Agent:’

How to capture the HTTP packets being transmitted between Webserver and Application server both GET & POST?

In cases, where you have check the HTTP traffic between webserver and application server.  you can use tcpdump to diagnose and troubleshoot the issue. It will be helpful for many middleware administrators. By this you can make sure that the  websphere proxy plugin (or) weblogic proxy plugin is forwarding the HTTP calls properly. In fact, you can also see to which managed server (or) node the call is going and what data is being transmitted Let’s suppose this is your environment

WebServer IP192.168.10.4
AppServer IP192.168.10.6
AppServer Listen Port18001

You can run the following command at the application server  to accomplish your requirement tcpdump -i any -s 0 -A ’tcp dst port 18001 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x3C21444F and host 192.168.10.4'

Application server Oriented examples

Setting the Context:  We are using weblogic application server for this post but it does not limit the scope of this post only to weblogic.  As I already mentioned earlier.  All you have to change is the interface and the port. Let me give some overview of the weblogic infrastructure been used for this post

Application Server productWeblogic Server 12c
Server Listening Port18001
Server listen address192.168.60.4
Application Full URLhttp://192.168.60.4:18001/TestWebService

| **Application Web Service EndPoint ** | http://192.168.60.4:18001/TestWebService/WS |

How to capture a Complete HTTP Transmission, incoming and outgoing Including both HTTP Request and Response.  Associated with a Single Client along with HTML data ( GET & POST ) on port 18001

All the examples we have given above can be used for weblogic with just a little change in port number. However. we are giving an example here. Here we are going to make a call from the client with up 192.168.60.1 to our TestWebService Application using GET and POST methods and capture the HTTP traffic data at the server end tcpdump -i enp0s8 -s 0 -A ’tcp dst port 18001 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x3C21444F and host 192.168.60.1' Sample Output Request from192.168.60.1 usingcurl -v aksarav@middlewareinventory:~$ curl -v “-H X-Forwarded-By: middlewareinventory” -X POST http://192.168.60.4:18001/TestWebService/

  • Trying 192.168.60.4…
  • TCP_NODELAY set
  • Connected to 192.168.60.4 (192.168.60.4) port 18001 (#0) > POST /TestWebService/ HTTP/1.1 > Host: 192.168.60.4:18001 > User-Agent: curl/7.54.0 > Accept: / > X-Forwarded-By: middlewareinventory > < HTTP/1.1 200 OK < Date: Sat, 28 Jul 2018 06:21:42 GMT < Accept-Ranges: bytes < Content-Length: 475 < Content-Type: text/html < Last-Modified: Sat, 28 Jul 2018 15:41:10 GMT < Sample WebService Application
    This is Sample WebService Application
  • Connection #0 to host 192.168.60.4 left intact

Response [root@mwiapp01 ~]# tcpdump -i enp0s8 -s 0 -A ’tcp dst port 18001 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x3C21444F and host 192.168.60.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 65535 bytes 02:49:12.424425 IP 192.168.60.1.60442 > mwiapp01.18001: Flags [P.], seq 558796881:558797017, ack 3750122298, win 4117, options [nop,nop,TS val 317440918 ecr 4685369], length 136 E…Z.@.@…..<…<…FQ!N.Q..S:……….. …..G~9POST /TestWebService/ HTTP/1.1 Host: 192.168.60.4:18001 User-Agent: curl/7.54.0 Accept: / X-Forwarded-By: middlewareinventory

02:49:12.426378 IP mwiapp01.18001 > 192.168.60.1.60442: Flags [P.], seq 1:171, ack 136, win 235, options [nop,nop,TS val 4685370 ecr 317440918], length 170 E…-.@.@…..<…<.FQ….S:!N…….&….. .G~:….HTTP/1.1 200 OK Date: Sun, 29 Jul 2018 06:49:12 GMT Accept-Ranges: bytes Content-Length: 475 Content-Type: text/html Last-Modified: Sat, 28 Jul 2018 15:41:10 GMT

02:49:12.426683 IP mwiapp01.18001 > 192.168.60.1.60442: Flags [P.], seq 171:646, ack 136, win 235, options [nop,nop,TS val 4685371 ecr 317440920], length 475 E…-.@.@…..<…<.FQ….S.!N…….W….. .G~;….Sample WebService Application

This is Sample WebService Application

SOAP Webservice Capture

How to capture the Webservice Traffic between Webservice endpoint deployed at weblogic and web service Client ( For example SOAP UI) ? Here is our web service End Point http://192.168.60.4:18001/TestWebService/WS Here we have to two methods.

  • **Hello –**Just to say hello to the name being passed in the request
  • GetSimilarWord –  To Get the string from the user and search the internet for the matching word and return a  JSON array

Let’s invoke the methods in SOAP UI and see what we see in tcpdump at the server end. [root@mwiapp01 ~]# tcpdump -i enp0s8 -s 0 -A ’tcp dst port 18001 and tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) » 2):4] = 0x30313136 and host 192.168.60.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 65535 bytes 03:25:45.147901 IP 192.168.60.1.60684 > mwiapp01.18001: Flags [P.], seq 2351207025:2351207604, ack 568016594, win 4104, options [nop,nop,TS val 319664919 ecr 6877781], length 579 E..w+9@.@…..<…<…FQ.$.q!.>………… …h.UPOST /TestWebService/WS HTTP/1.1 Content-Type: text/xml cache-control: no-cache Postman-Token: 8d979543-94ff-4a32-9570-975bfa48d644 User-Agent: PostmanRuntime/7.1.1 Accept: / Host: 192.168.60.4:18001 accept-encoding: gzip, deflate content-length: 291 Connection: keep-alive

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/" xmlns:mid=“https://middlewareinventory.com/"> soapenv:Header/ soapenv:Body mid:hello AKSarav </mid:hello> </soapenv:Body> </soapenv:Envelope> 03:25:45.152523 IP mwiapp01.18001 > 192.168.60.1.60684: Flags [P.], seq 1:124, ack 579, win 245, options [nop,nop,TS val 6878096 ecr 319664919], length 123 E….:@.@…..<…<.FQ..!.>..$…………. ..HTTP/1.1 200 OK Date: Sun, 29 Jul 2018 07:25:45 GMT Transfer-Encoding: chunked Content-Type: text/xml; charset=utf-8

03:25:45.152909 IP mwiapp01.18001 > 192.168.60.1.60684: Flags [P.], seq 124:410, ack 579, win 245, options [nop,nop,TS val 6878097 ecr 319664923], length 286 E..R.;@.@…..<…<.FQ..!.?M.$…………. ..0116 <S:Envelope xmlns:S=“http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns0:helloResponse xmlns:ns0=“https://middlewareinventory.com/">Hello AKSarav ! Welcome to middlewareinventory</ns0:helloResponse></S:Body></S:Envelope>

^C 3 packets captured 3 packets received by filter 0 packets dropped by kernel

You can see the SOAP XML response from the server being returned to the SOAP UI. For those who are not familiar with SOAP UI. Use Postman as shown below.

So far we have discussed various usage and examples of TCPDUMP.  But I am certain there might be a whole lot of requirements and examples that I may have missed in this post. Instead of using the tcpdump command precisely to perform/capture request (or) response. You could also record the tcpdump session ( take a dump of tcpdump) and analyze, In fact, it is more trustworthy and easy rather trying to write command with ASCII code matching.

How to record a TCPDUMP Session (or) Capture packets with tcpdump

To record the tcpdump session, you can use the following command Note*: here I have used any as an interface to capture all the packets across all the channels/interfaces available in my server tcpdump -i any -s 0 -X -w /tmp/tcpdump.pcap pcap is a widely accepted extension for the tcpdump output.

How to read the TCPDUMP recorded session (or) packet capture – pcap file

tcpdump -A -r /tmp/tcpdump.pcap This way. you would be able to read the recorded session and it will offer more information than the ASCII matching commands. You can also use commands like less for better readability and search. tcpdump -A -r /tmp/tcpdump.pcap|less

The Wireshark There is a wonderful Sniffing tool (network packet analyzer) available in the market named Wireshark. You can just import the pcap file and process your tcpdump output like never before. You will just be amazed by the details and options that Wireshark provides. I Guarantee it. Look at this sample shows the detailed information and the XML response it has sent to the Client. Ohh man. It took a week to complete this post and I still feel like I have missed so many things. let me write another article dedicatedly for Wireshark and its capabilities. Any questions, Feedback, Scoldings. Please use the comments section. Would be really happy to see that someone is showing interest.  It’s pretty lonely here. So comment anyway See ya Sarav AK


最后修改 October 4, 2023: 合并 commit (98ba273b)