In the first part of the blog on the TWS API of Interactive Brokers, I have dealt with the Installation of TWS API on a Windows machine for Anaconda distro.
In the second part of the blog, titled Fundamentals of TWS API, I have dealt with the essentials to get started.
Now in this third part I seek to explore the concepts like using a scanner, retrieving the OHLCV data and writing it to a CSV file, Hedging a Futures Contract with Options and many others including a fully functional sample trading system.
Table of Contents :
Optionable Contracts in NSE using the Scanner
Let's get to the point straight - The TWS API is just an interface to the TWS. If you are having problems defining a scanner viz the API , always make sure you can create a similar scanner using the TWS or the Mosaic Market Scanner.
To get a better understanding, take a look at the screenshot of the TWS scanner
Here the instrument is "Asia Stocks", the Location is "India" , the filter is "Has Options is" and the Parameter is "Shortable Shares"
Now let's look at how we can code these criteria in Python using the TWS API and get the desired results.
For this the first thing we need to do is find out what are the scanner parameters. We can do this via reqScannerParameters. (Note that not all of the returned parameters to scannerParameters are available from the API scanners)
# Request the scanner parameters
client.reqScannerParameters()
The callback to the above request is scannerParameters. A string containing all available XML formatted parameters will then be returned. The XML file size was around 2 MB .
def scannerParameters(self, xml: str):
''' Callback for reqScannerParameters '''
super().scannerParameters(xml)
#open('log/scanner.xml', 'w').write(xml)
open('scanner.xml', 'w').write(xml)
print("ScannerParameters received."
Open the XML file and peruse it. Then you will find the right parameters to be used while creating the ScannerSubscription object to retrieve NSE stocks.
Accessing the market scanner programmatically involves the following steps:
- Create a scanner subscription using ScannerSubscription
- Request a scanner subscription by calling reqScannerSubscription.
- Access the scanner results using the callback scannerData.
- Stop the scanner by calling cancelScannerSubscription.
The scannerData callback will be called once for each desired result up to a maximum of 50. When the data transmission is complete, the scannerDataEnd callback function will be called.
Firstly we need to create an instance of ScannerSubscription. There are various fields that can be set but the following three are the most important:
- Instrument
- locationCode
- scanCode
The below code is a typical example of what I would set for the NSE stocks:
# Create the object ScannerSubscription scanSub = ScannerSubscription() # Defines a market scanner request scanSub.instrument = 'STOCK.HK' scanSub.locationCode = 'STK.HK.NSE' scanSub.scanCode = 'ALL_SYMBOLS_ASC'
We get only 50 securities in one call to reqScannerSubscription . So, to setup simple filters, we have flexibility. As an example to ensure that we get the stocks whose volume is above 50,000, we can set the ScannerSubscription's aboveVolume field to 50000.
scanSub.aboveVolume = '50000'
There are a host of permutations and combinations that you can set thee fields to and these are available in the XML file. For more examples, open up the ScannerSubscriptionSamples.py program in C:\TWS API\samples\Python\Testbed and check them.
The ScannerSubscription Class is in the scanner.py program located at C:\TWS API\source\pythonclient\ibapi. This Class does not have any methods of its own and it just holds data for the desired subscription. Our Python program passes this data to IB by calling the reqScannerSubscription.
reqScannerSubscription(int reqId, ScannerSubscription subscription, List< TagValue > scannerSubscriptionOptions, List< TagValue > ScannerSubscriptionFilterOptions )
The third argument "scannerSubscriptionOptions" is reserved for internal use by IB. The last argument "ScannerSubscriptionFilterOptions" is of importance to us. This is how we set additional filter conditions. This accepts a container of name = value pairs . Here the name identifies the nature of the filter and the value sets the value that the criteria is checked against.
Now let's take a look at the screen shot of the TWS scanner above. Here we have used a filter "Has Options" and set this to Yes. Now if we open the XML file received from scannerParameters callback and make a search, this is what we find:
Highlighted in the above screenshot is the <displayName> tag. Above that is the <AbstractField> tag. Just below is the <Code> tag and this contains the filter names. If you search through the XML file, you will find many filter names and some are given below:
- hasOptionsIs
- optVolumeAbove
- impVolatAbove
# Create the object ScannerSubscription
scanSub = ScannerSubscription() # Defines a market scanner request
scanSub.instrument = 'STOCK.HK'
scanSub.locationCode = 'STK.HK.NSE'
scanSub.scanCode = 'ALL_SYMBOLS_ASC'
# Set additional filter criteria
tagvalues = []
tagvalues.append(TagValue('avgOptVolumeAbove', '0'))
tagvalues.append(TagValue('hasOptionsIs', 'Yes'))
print(tagvalues)
# Request the scanner subscription
client.reqScannerSubscription(7, scanSub, [], tagvalues)
When I ran the above code, the callback did not give me the desired results. So basically the "hasOptionsIs" does not work. Then I tried with tagvalue "avgOptVolumeAbove" and that also didn’t work. That’s when I realised that I will have to seek an alternative path to try and get the optionable stocks. A search on the TWS API group reveals that these tagvalues have been documented in the XML file but are yet to be incorporated in the Python based API. So yeah, this could possibly be there in the next/future versions of the TWS API. (This program has been tested on the API version 9.79 with release date 05 Feb 2020).
If you click on the corresponding link to NSE , you will find that it gives you the IB symbol and the actual NSE Symbol. In the case of Baja Auto, the IB symbol is "BAJAJ-AUT" while the actual NSE symbol is "BAJAJ-AUTO". Be aware of this.
# Create the object ScannerSubscription scanSub = ScannerSubscription() # Defines a market scanner request scanSub.instrument = 'SSF.HK' # SSF = Single Stock Futures scanSub.locationCode = 'SSF.HK.NSE' scanSub.scanCode = 'ALL_SYMBOLS_ASC'
This is the output in my terminal.
Now if you watch closely, each SSF is repeated three times. This is becos NSE has three future contracts for a given instrument at any given point of time - Near, Next and Far.
So now, our goal is to just get all the available contracts and then ensure that each symbol is unique in the list. This will give us the complete list of Optionable Contracts. We can then cross verify this list with the official NSE list of Underlyings & Underlying Info .
But unfortunately, at the moment, using the TWS API, we can get only 50 rows from each scanner request. That’s how it is with the TWS also. Also at the moment, You can't have more than 10 scanner subscriptions running same time.
So yeah, what we can do is run the scanner multiple times with different scan parameters and collect the output from the call backs. Let us tweak the scan parameters once again to get better results. Check the code below:
def main():
# Create the client and connect to TWS
client = StockScanner('127.0.0.1', 7497, 7)
time.sleep(3)
# Create the object ScannerSubscription
scanSub = ScannerSubscription() # Defines a market scanner request
scanSub.instrument = 'STOCK.HK'
scanSub.locationCode = 'STK.HK.NSE'
# scanSub.scanCode = 'ALL_SYMBOLS_ASC'
scanSub.scanCode = 'TOP_TRADE_COUNT'
# Set additional filter criteria
tagvalues = []
tagvalues.append(TagValue('hasOptionsIs', 'true'))
tagvalues.append(TagValue("usdPriceAbove", "0"))
tagvalues.append(TagValue("usdPriceBelow", "2"))
print(tagvalues)
# Request the scanner subscription
client.reqScannerSubscription(7, scanSub, [], tagvalues)
# Disconnect from TWS
time.sleep(5)
client.disconnect() 
Check out the Code for this on GitHub
Note that at the time of writing, I am using the TWS API version 9.79 with release date 05 Feb 2020. Maybe for latter versions we can Hope that things will be better !!!
Work Around Solution: So what we can do is check out the link for Single Stock Futures (SSF's) in NSE. Copy them into an Excel or CSV file or run a webscraping program . And use this list as the starting point for all your strategies that require "Optionable Stocks".
Write OHLCV data to Pandas and CSV files
Lets go through the code straight.
def __init__(self, addr, port, client_id):
EClient.__init__(self, self)
# Initialize properties
self.symbols = {'ACC':'44652144','INFY':'44652017', 'WIPRO':'44652030'} # IB symbol could be different from NSE symbol, so use conId
self.candle_dict = {}
The first thing to realize is that the Symbols that IB uses internally is not the same as the symbols used by the National Stock Exchange(NSE).
If you go to the market watchlist on TWS, click on a Symbol --> Financial Instrument Info --> Description , in the popup windows, you get all the basic information about the stock. That includes ISIN number, which is used by the NSE. I would have loved to plug this into the "value" in the key-value pair of the self.symbols dictionary.
Now in TW go to Symbol --> Financial Instrument Info --> Details. It will open up a new tab in your browser where you have more details. Check out for "Contract Identifiers" and you will find that whether we are referring to a Futures instrument or its underlying, the conId is a constant. So let us stick to that instead of ISIN number.
I have investigated and it appears that there are two ways of getting the conId of a given stock:
1) Take this info from the IB website.
@iswrapper
def historicalData(self, req_id, bar):
''' Callback to reqHistoricalData '''
# Add the stock prices to the dictionary
self.candle_dict['Date'].append(bar.date)
self.candle_dict['Open'].append(bar.open)
self.candle_dict['High'].append(bar.high)
self.candle_dict['Low'].append(bar.low)
self.candle_dict['Close'].append(bar.close)
self.candle_dict['Volume'].append(bar.volume) # whatToShow in reqHistoricalData should be TRADES
def main():
# Create the client and connect to TWS
client = ReadTicker('127.0.0.1', 7497, 7)
time.sleep(3) #Sleep interval to allow time for connection to server
# Get expiration dates for contracts
for symbol in client.symbols:
# Define the contract
contract = Contract()
contract.symbol = symbol
contract.secType = 'STK'
contract.exchange = 'NSE'
contract.conId = client.symbols[symbol]
print('Contract ID for symbol {} is: {}'.format(symbol, client.symbols[symbol])) #take conId from IB website or contractDetails callback
contract.currency = "INR"
# Initialize the candle data dictionary
for v in ['Date','Open', 'High', 'Low', 'Close', 'Volume']:
client.candle_dict[v] = []
# Request OHLCV data
now = datetime.now().strftime("%Y%m%d %H:%M:%S")
client.reqHistoricalData(4, contract, now, '2 D', '5 mins', 'TRADES', False, 1, False, [])
time.sleep(3) # Allow enough time for data to be returned.
df = pd.DataFrame(client.candle_dict, columns=['Date', 'Open', 'High', 'Low', 'Close', 'Volume'])
df['Date'] = pd.to_datetime(df['Date']) # Convert to pandas DateTime format
# print(df.tail())
# print(df.columns)
df.to_csv(symbol + '.csv', encoding='utf-8', index=False)
client.candle_dict.clear()
# Disconnect from TWS
client.disconnect() Web Scrap IB site for Single Stock Futures (SSF)
This is the URLwhere Interactive broker's has data about all the Single Stock Futures(SSF).
Let's look at the Web Scraping option's to get hold of all the SSF's. We have basically three modules that we can consider - Scrapy, Selenium and Beautiful Soup.
In this specific case, we are primarily concerned about getting the contents of the table below. And this table spreads across to two pages. So our job narrows down to getting the contents of the table from two different URL's
Since it’s a pretty straightforward case of merely extracting some information from a website, I feel that Selenium and Scrapy will be too much of an overkill. So let's settle down for Beautiful Soup. In conjunction with requests module, it should do the trick.
Step1: Identify the element to extract.
On the Chrome browser, right click on Inspect and hover over the elements until you get to the point where the entire table that we want is covered . Now you know that the element is a table element in the HTML document. Or another simple method is right click on the web page and click on "View Page Source". Scan through the page source code until you find the Table tag . (The <table> tag defines an HTML table).
In case you want to learn the finer details of extracting the HTML elements, Tags, CSS etc. by using Beautiful Soup, check out the comprehensive Beautiful Soup documentation
Step2: Add header and URL to the requests.
This helps to create a request to the website. The header is sometimes necessary and useful to spoof your request so that it looks like it comes from a legitimate browser. Quite a few financial sites actually block requests that do not have a legit header.
requests_headers = {'User-Agent': 'Mozilla/5.0 6(Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}
url1 = "https://www.interactivebrokers.com/en/index.php?f=2222&exch=nse&showcategories=SSF" # first page has 100 ssf url2 = "https://www.interactivebrokers.com/en/index.php?f=2222&exch=nse&showcategories=SSF&p=&cc=&limit=100&page=2"
Step4: Pass to requests module.
When we pass any URL, the requests library will make a GET request to a web server, which will download the HTML contents of a given web page for us. After running our request, we get a Response object. This object has a status_code property, which indicates if the page was downloaded successfully. A status_code of 200 means that the page downloaded successfully. We won’t fully dive into status codes here, but a status code starting with a 2 generally indicates success, and a code starting with a 4 or a 5 indicates an error. We can print out the HTML content of the page using the content property:
response = requests.get(url, headers=requests_headers)
print('\nThe URL: {}'.format(response.url))
print('The Status Code from Requests: {}'.format(response.status_code)) # print status code
Step5: Initiate BS and list element to extract all the rows in the table
We can use the BeautifulSoup library to parse this document. We have to create an instance of the BeautifulSoup class to parse our document. Next if we watch the structure of the webpage carefully, we will notice that there are thee tables and the third one is what we want to scrap.
soup = BeautifulSoup(response.content, 'lxml') # Parse the HTML as a string
table = soup.find_all('table')[2] # Grab the third table on the webpage
df = pd.DataFrame(columns=range(0,4), index = [0])
for row_marker, row in enumerate(table.find_all('tr')):
if data==True:
columns = row.find_all(['td']) # Capture only the Table Data Cells.
else:
columns = row.find_all(['th']) # Capture only the Table Header Cells.
try:
df.loc[row_marker] = [column.get_text() for column in columns]
except ValueError:
# It's a safe way to handle when [column.get_text() for column in columns] is empty list.
continue
df_table_header = get_table_data(url1, data=False)
if df_table_header.iloc[0,0] == 'IB Symbol': # confirm 'IB Symbol' content of first row , then change next column
df_table_header.iloc[0,1] = 'Product Description' # Original description is too long. Cut it short
else:
print ("WARNING !!!.Some Change in the Header Data of IB webitse. ***CHECK ALL DATA***")
df_table_data_P1 = get_table_data(url1, data=True)
df_table_data_P2 = get_table_data(url2, data=True)
df_final = pd.concat([df_table_header, df_table_data_P1, df_table_data_P2])
df_final = df_final.rename(columns=df_final.iloc[0]) # Rename first row to column header
df_final = df_final.dropna() # Drop rows where atleast one element is missing
df_final = df_final[df_final.Symbol != 'Symbol'] # if there is row with text 'symbol', exclude that
df_final = df_final[df_final.Symbol != 'NIFTY'] # If there is row with 'NIFTY50', exclude that
df_final = df_final[df_final.Symbol != 'BANKNIFTY'] # If there is row with 'BANKNIFTY', exclude that
df_final = df_final[df_final.Symbol != 'USDINR'] # If there is row with 'USDINR', exclude that
df_final = df_final[df_final.Symbol != 'EURINR'] # If there is row with 'EURINR', exclude that
df_final = df_final[df_final.Symbol != 'GBPINR'] # If there is row with 'GBPINR', exclude that
df_final = df_final[df_final.Symbol != 'JPYINR'] # If there is row with 'JPYINR', exclude that
df_final = df_final.reset_index(drop=True)
df_final.to_csv('scraped_data.csv', index=False) # dont write row names
Creating a Stock Universe
# Define a Contract
contract = Contract()
contract.symbol = 'INFY'
contract.conId = '437440510'
contract.secType = 'STK'
contract.currency = 'INR'
contract.primaryExchange = "NSE"
Check out this url for the conID of stock symbol Infyimport collections
class CreateStockUniverse(EWrapper, EClient):
''' Class that Serves as the Client and the Wrapper '''
def __init__(self, addr, port, client_id):
EWrapper.__init__(self)
EClient.__init__(self, self)
# Initialize properties
df_scraped_data = pd.read_csv('scraped_data.csv') # Read csv into pandas DataFrame
self.df_scraped_data = df_scraped_data
self.conId_dq = collections.deque() # Store Contract ID in deque
# Connect to TWS API
self.connect(addr, port, client_id)
# Launch the client thread
thread = threading.Thread(target=self.run)
thread.start()
@iswrapper
def contractDetails(self, reqId, details):
''' Callback from '''
print('Details from contractDetails Callback follows:')
print('Market Name: {}'.format(details.marketName)) # symbol=TATACONSU, marketname=TATACONSUM
print('Long Name: {}'.format(details.longName))
print('Contract ID: {}'.format(details.contract.conId))
# print('Time Zone for the Product: {}'.format(details.timeZoneId))
# print(details)
self.conId_dq.append(details.contract.conId)
conId_list = list(client.conId_dq)
df_conId = pd.DataFrame([conId_list]).transpose()
df_conId.columns = ['Contract ID']
df_final = client.df_scraped_data.join(df_conId)
print ('Length of df_scraped_data: {}'.format(len(client.df_scraped_data)))
print ('Length of df_conId: {}'.format(len(df_conId)))
print('Length of df_final: {}'.format(len(df_final)))
df_final.to_csv('stock_universe_web_scraping'+ '.csv', encoding='utf-8', index=False)
No comments:
Post a Comment