Understanding how this works is an important piece of learning to analyze data efficiently and effectively with Python. But what this type flexibility also points to is the fact that Python variables are more than just their value; they also contain extra information about the type of the value. This means that every Python object is simply a cleverly disguised C structure, which contains not only its value, but other information as well.
Looking through the Python 3. Figure Notice the difference here: a C integer is essentially a label for a position in memory whose bytes encode an integer value. This extra information in the Python integer structure is what allows Python to be coded so freely and dynamically. All this additional information in Python types comes at a cost, however, which becomes especially apparent in structures that combine many of these objects.
The standard mutable multielement container in Python is the list. In the special case that all variables are of the same type, much of this information is redundant: it can be much more efficient to store data in a fixed-type array. The difference between a dynamic-type list and a fixed-type NumPy-style array is illustrated in Figure The Python list, on the other hand, contains a pointer to a block of pointers, each of which in turn points to a full Python object like the Python integer we saw earlier.
Again, the advantage of the list is flexibility: because each list element is a full structure containing both data and type information, the list can be filled with data of any desired type. The difference between C and Python lists Fixed-Type Arrays in Python Python offers several different options for storing data in efficient, fixed-type data buffers. The built-in array module available since Python 3. Much more useful, however, is the ndarray object of the NumPy package. If types do not match, NumPy will upcast if possible here, integers are upcast to floating point : In[9]: np.
Here are several examples: In[12]: Create a length integer array filled with zeros np. Because NumPy is built in C, the types will be familiar to users of C, Fortran, and other related languages. The standard NumPy data types are listed in Table Note that when constructing an array, you can specify them using a string: np. While the types of operations shown here may seem a bit dry and pedantic, they comprise the building blocks of many other examples used throughout the book.
Get to know them well! This means, for example, that if you attempt to insert a floating-point value to an integer array, the value will be silently truncated. In this case, the defaults for start and stop are swapped. For example: In[24]: x2 Out[24]: array [[12, 5, 2, 4], [ 7, 6, 8, 8], [ 1, 6, 7, 7]] In[25]: x2[:2, :3] two rows, three columns Out[25]: array [[12, 5, 2], [ 7, 6, 8]] In[26]: x2[:3, ] all rows, every other column Out[26]: array [[12, 2], [ 7, 8], [ 1, 7]] Finally, subarray dimensions can even be reversed together: In[27]: x2[, ] Out[27]: array [[ 7, 7, 6, 1], [ 8, 8, 6, 7], [ 4, 2, 5, 12]] Accessing array rows and columns.
One commonly needed routine is accessing single rows or columns of an array. This is one area in which NumPy array slicing differs from Python list slicing: in lists, slices will be copies. Creating copies of arrays Despite the nice features of array views, it is sometimes useful to instead explicitly copy the data within an array or a subarray.
The most flexible way of doing this is with the reshape method. Where possible, the reshape method will use a no-copy view of the initial array, but with noncontiguous memory buffers this is not always the case. Another common reshaping pattern is the conversion of a one-dimensional array into a two-dimensional row or column matrix. Array Concatenation and Splitting All of the preceding routines worked on single arrays.
Concatenation of arrays Concatenation, or joining of two arrays in NumPy, is primarily accomplished through the routines np. Splitting of arrays The opposite of concatenation is splitting, which is implemented by the functions np. The related functions np. Computation on NumPy Arrays: Universal Functions Up until now, we have been discussing some of the basic nuts and bolts of NumPy; in the next few sections, we will dive into the reasons that NumPy is so important in the Python data science world.
Computation on NumPy arrays can be very fast, or it can be very slow. It then introduces many of the most common and useful arithmetic ufuncs available in the NumPy package. This is in part due to the dynamic, interpreted nature of the language: the fact that types are flexible, so that sequences of operations cannot be compiled down to efficient machine code as in languages like C and Fortran.
Each of these has its strengths and weaknesses, but it is safe to say that none of the three approaches has yet surpassed the reach and popularity of the standard CPython engine.
A straightforward approach might look like this: In[1]: import numpy as np np. But if we measure the execution time of this code for a large input, we see that this operation is very slow, perhaps surprisingly so!
It turns out that the bottleneck here is not the operations themselves, but the type-checking and function dispatches that CPython must do at each cycle of the loop. Introducing UFuncs For many types of operations, NumPy provides a convenient interface into just this kind of statically typed, compiled routine.
This is known as a vectorized operation. You can accomplish this by simply performing an operation on the array, which will then be applied to each element. This vectorized approach is designed to push the loop into the compiled layer that underlies NumPy, leading to much faster execution.
Ufuncs are extremely flexible—before we saw an operation between a scalar and an array, but we can also operate between two arrays: In[5]: np.
Any time you see such a loop in a Python script, you should consider whether it can be replaced with a vectorized expression. Table The basic np. A look through the NumPy documentation reveals a lot of interesting functionality.
Another excellent source for more specialized and obscure ufuncs is the submodule scipy. If you want to compute some obscure mathematical function on your data, chances are it is implemented in scipy. Advanced Ufunc Features Many NumPy users make use of ufuncs without ever learning their full set of features.
Specifying output For large calculations, it is sometimes useful to be able to specify the array where the result of the calculation will be stored. Aggregates For binary ufuncs, there are some interesting aggregates that can be computed directly from the object. A reduce repeatedly applies a given operation to the elements of an array until only a single result remains. Outer products Finally, any ufunc can compute the output of all pairs of two different inputs using the outer method. Another extremely useful feature of ufuncs is the ability to operate between arrays of different sizes and shapes, a set of operations known as broadcasting.
Summing the Values in an Array As a quick example, consider computing the sum of all values in an array. Multidimensional aggregates One common type of aggregation operation is an aggregate along a row or column. Similarly, we can find the maximum value within each row: In[12]: M. The axis keyword specifies the dimension of the array that will be collapsed, rather than the dimension that will be returned. Some of these NaN-safe functions were not added until NumPy 1.
Table provides a list of useful aggregation functions available in NumPy. We may also wish to compute quantiles: In[16]: print "25th percentile: ", np.
Broadcasting is simply a set of rules for applying binary ufuncs addition, subtraction, multiplication, etc. We can similarly extend this to arrays of higher dimension. While these examples are relatively easy to understand, more complicated cases can involve broadcasting of both arrays.
The geometry of these examples is visualized in Figure Visualization of NumPy broadcasting The light boxes represent the broadcasted values: again, this extra memory is not actually allocated in the course of the operation, but it can be useful conceptually to imagine that it is.
Used with permission. The shapes of the arrays are: M. How does this affect the calculation? But this is not how the broadcasting rules work! That sort of flexibility might be useful in some cases, but it would lead to potential areas of ambiguity. Centering an array In the previous section, we saw that ufuncs allow a NumPy user to remove the need to explicitly write slow Python loops.
Broadcasting extends this ability. Imagine you have an array of 10 observations, each of which consists of 3 values. Plotting a two-dimensional function One place that broadcasting is very useful is in displaying images based on two- dimensional functions. In NumPy, Boolean masking is often the most efficient way to accomplish these types of tasks. Example: Counting Rainy Days Imagine you have a series of data that represents the amount of precipitation each day for a year in a given city.
What is the average precipitation on those rainy days? How many days were there with more than half an inch of rain? Digging into the data One approach to this would be to answer these questions by hand: loop through the data, incrementing a counter each time we see values in some desired range. The result of these comparison operators is always an array with a Boolean data type. Working with Boolean Arrays Given a Boolean array, there are a host of useful operations you can do.
Another way to get at this information is to use np. For example: In[22]: are all values in each row less than 8? These have a different syntax than the NumPy versions, and in particular will fail or produce unintended results when used on multidimensional arrays.
Be sure that you are using np. But what if we want to know about all days with rain less than four inches and greater than one inch? For example, we can address this sort of compound question as follows: In[23]: np. Here are some examples of results we can compute when combining masking with aggregations: In[25]: print "Number days without rain: ", np.
A more powerful pattern is to use Boolean arrays as masks, to select particular subsets of the data themselves. We are then free to operate on these values as we wish.
When would you use one versus the other? In Python, all nonzero integers will evaluate as True. For Boolean NumPy arrays, the latter is nearly always the desired operation. Fancy Indexing In the previous sections, we saw how to access and modify portions of arrays using simple indices e. For example: In[8]: row[:, np. Modifying Values with Fancy Indexing Just as fancy indexing can be used to access parts of an array, it can also be used to modify parts of an array.
For example: 82 Chapter 2: Introduction to NumPy www. The result, of course, is that x[0] contains the value 6. Why is this not the case? With this in mind, it is not the augmentation that happens multiple times, but the assignment, which leads to the rather nonintuitive results.
So what if you want the other behavior where the operation is repeated? For this, you can use the at method of ufuncs available since NumPy 1. Another method that is similar in spirit is the reduceat method of ufuncs, which you can read about in the NumPy documentation.
Example: Binning Data You can use these ideas to efficiently bin data to create a histogram by hand. For example, imagine we have 1, values and would like to quickly find where they fall within an array of bins.
We could compute it using ufunc. A histogram computed by hand Of course, it would be silly to have to do this each time you want to plot a histogram. This is why Matplotlib provides the plt. To compute the binning, Matplotlib uses the np. How can this be? If you dig into the np. Sorting Arrays Up to this point we have been concerned mainly with tools to access and operate on array data with NumPy. This section covers algorithms related to sorting values in NumPy arrays.
All are means of accomplishing a similar task: sorting the values in a list or array. Fortunately, Python contains built-in sorting algorithms that are much more efficient than either of the simplistic algorithms just shown.
Fast Sorting in NumPy: np. By default np. To return a sorted version of the array without modifying the input, you can use np. NumPy provides this in the np.
Within the two partitions, the elements have arbitrary order. Similarly to sorting, we can partition along an arbitrary axis of a multidimensional array: In[13]: np.
Finally, just as there is a np. With the pairwise square-distances converted, we can now use np. We can do this with the np. Visualization of the neighbors of each point Each point in the plot has lines drawn to its two nearest neighbors. At first glance, it might seem strange that some of the points have more than two lines coming out of them: this is due to the fact that if point A is one of the two nearest neighbors of point B, this does not necessarily imply that point B is one of the two nearest neighbors of point A.
Although the broadcasting and row-wise sorting of this approach might seem less straightforward than writing a loop, it turns out to be a very efficient way of operating on this data in Python. You might be tempted to do the same type of operation by manually looping through the data and sorting each set of neighbors individually, but this would almost certainly lead to a slower algorithm than the vectorized version we used.
Big-O Notation Big-O notation is a means of describing how the number of operations required for an algorithm scales as the input grows in size. Far more common in the data science world is a less rigid use of big-O notation: as a general if imprecise description of the scaling of an algorithm.
Big-O notation, in this loose sense, tells you how much time your algorithm will take as you increase the amount of data. For our purposes, the N will usually indicate some aspect of the size of the dataset the number of points, the number of dimensions, etc.
Notice that the big-O notation by itself tells you nothing about the actual wall-clock time of a computation, but only about its scaling as you change N. But for small datasets in particular, the algorithm with better scaling might not be faster. Creating Structured Arrays Structured array data types can be specified in a number of ways. Earlier, we saw the dictionary method: In[10]: np.
The next character specifies the type of data: characters, bytes, ints, floating points, and so on see Table The last character or characters represents the size of the object in bytes. For example, you can create a type where each element contains an array or matrix of values. Why would you use this rather than a simple multidimensional array, or perhaps a Python dictionary? On to Pandas This section on structured and record arrays is purposely at the end of this chapter, because it leads so well into the next package we will cover: Pandas.
Pandas is a newer package built on top of NumPy, and provides an efficient implementation of a DataFrame. As well as offering a convenient storage interface for labeled data, Pandas implements a number of powerful data operations familiar to users of both database frameworks and spreadsheet programs. In this chapter, we will focus on the mechanics of using Series, DataFrame, and related structures effectively. We will use examples drawn from real datasets where appropriate, but these examples are not necessarily the focus.
Details on this installation can be found in the Pandas documentation. If you followed the advice outlined in the preface and used the Anaconda stack, you already have Pandas installed.
Once Pandas is installed, you can import it and check the version: In[1]: import pandas pandas. For example, to display all the contents of the pandas namespace, you can type this: In [3]: pd. Introducing Pandas Objects At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy structured arrays in which the rows and columns are identified with labels rather than simple integer indices.
As we will see during the course of this chapter, Pandas provides a host of useful tools, methods, and functionality on top of the basic data structures, but nearly everything that follows will require an understanding of what these structures are.
Series [0. The values are simply a familiar NumPy array: In[3]: data. For example, the index need not be an integer, but can consist of values of any desired type. A dictionary is a structure that maps arbitrary keys to a set of arbitrary values, and a Series is a structure that maps typed keys to a set of typed values. This typing is important: just as the type-specific compiled code behind a NumPy array makes it more efficient than a Python list for certain operations, the type information of a Pandas Series makes it much more efficient than Python dictionaries for certain operations.
For example, data can be a list or NumPy array, in which case index defaults to an integer sequence: In[14]: pd. DataFrame as a generalized NumPy array If a Series is an analog of a one-dimensional array with flexible indices, a DataFrame is an analog of a two-dimensional array with both flexible row indices and flexible column names.
Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a DataFrame as a sequence of aligned Series objects. DataFrame as specialized dictionary Similarly, we can also think of a DataFrame as a specialization of a dictionary.
Where a dictionary maps a key to a value, a DataFrame maps a column name to a Series of column data. For a DataFrame, data['col0'] will return the first column. From a single Series object.
Any list of dictionaries can be made into a DataFrame. DataFrame data Out[24]: a b 0 0 0 1 1 2 2 2 4 Even if some keys in the dictionary are missing, Pandas will fill them in with NaN i. As we saw before, a DataFrame can be constructed from a dictionary of Series objects as well: In[26]: pd. Given a two-dimensional array of data, we can create a DataFrame with any specified column and index names. If omitted, an integer index will be used for each: In[27]: pd. DataFrame np.
This Index object is an interesting structure in itself, and it can be thought of either as an immutable array or as an ordered set technically a multiset, as Index objects may contain repeated values. Those views have some interesting consequences in the operations available on Index objects. Index as ordered set Pandas objects are designed to facilitate operations such as joins across datasets, which depend on many aspects of set arithmetic.
These included indexing e. Data Selection in Series As we saw in the previous section, a Series object acts in many ways like a one- dimensional NumPy array, and in many ways like a standard Python dictionary. Examples of these are as follows: In[7]: slicing by explicit index data['a':'c'] Out[7]: a 0. Notice that when you are slicing with an explicit index i. Indexers: loc, iloc, and ix These slicing and indexing conventions can be a source of confusion.
For example, if your Series has an explicit integer index, an indexing operation such as data[1] will use the explicit indices, while a slicing operation like data[] will use the implicit Python-style index. First, the loc attribute allows indexing and slicing that always references the explicit index: In[14]: data. For certain simulations, it may be necessary to use the same random sequence. Consult the documentation on how to set the seed of the random.
Use the 4-mers as keys and the number of appearances as values in the dictionary. The first entry sys. Our program will print out the list of arguments; will exit if no argument is provided.
Example usage: args. The function then reads a line from input, converts it to a string stripping a trailing newline , and returns that. This program first prompts the user to type an NCBI sequence number and then echos it. Second, it illustrates the casting of string input with int for use in arithmetical operations by calculating the sum of two user-typed numbers.
The number of bps is: It is possible to write programs that handle selected exceptions with a try-except statement. The try statement works as follows: 1.
The statement s between the try and except keywords is executed. If no exception happens, the except block is skipped and execution of the try statement is finished. If an exception occurs during execution of the try clause, the rest of the clause is skipped. Then if its type matches the exception named after the except keyword, the except clause is executed, and then execution continues after the try statement. An unhandled exception stops the execution. Optional else clause must follow all except clauses.
It is useful for code that must be executed if the try clause does not raise an exception. The code below illustrates the usage of try-except statement in handling exceptions, e.
The program asks for an input, repeatedly, if an exception is raised. If no exception is raised, the program breaks out of the infinite while loop by executing the else clause and finishes with a message. Try again It must be followed by the function name and the parenthesized list of formal parameters.
The statements that form the body of the function start at the next line, and must be indented. Variables in a function are local to that function. The first statement of the function body can optionally be a string literal enclosed in triple quotes ''' The user-defined function below takes a DNA segment as a string and returns the corresponding RNA segment as a string.
Utilities - Exercises V. Keyboard inputs: Write a Python program that asks the user to enter a DNA sequence from the keyboard and captures the sequence in a variable. Next, your program should ask the user for a second shorter DNA sequence to be entered. Now, your program should report if the second sequence is a substring of the first string.
Arguments as input: Write a Python program that takes two strings as command-line arguments and reports if the second argument is a substring of the first argument. What does it do? Consult the documentation on how to use the sys. What does the Python code below do? Modify the input. Traceback most recent call last : File "input. A function for growth of Florida sandhill cranes: In the range episode, we encountered the following problem: The population of Florida sandhill cranes in the Okefenokee swamp, under the current environmental conditions, grows by 1.
If we start with a population of birds, how big will the population be after 28 years? Now, create a Python function that, given an initial population size, growth rate, and a number of years, can calculate and return the Florida sandhill crane population size after that time period.
Modules: Python has a way to put function definitions in a file and use them in a script. Such a file is called a module. Definitions from a module can be imported into other modules or into the main program.
Consult the Python tutorial Modules for further information. Write a Python program to do the following: 1. Take a DNA segment, 3. Check if you get the original DNA segment back. Make sure that your function handles both upper and lowercases. Reverse complement: Write a Python function that takes a DNA sequence as a string and returns its reverse complement. Two parameters: Write a Python function that takes a DNA sequence as a string and a nucleotide and returns the number of occurrences of the nucleotide in the sequence.
Some PDF documents have an encryption feature that will keep them from being read until whoever is opening the document provides a password. Enter the following into the interactive shell with the PDF you downloaded, which has been encrypted with the password rosebud :.
PdfFileReader open 'encrypted. If given the wrong password, the decrypt function will return 0 and getPage will continue to fail. After your program terminates, the file on your hard drive remains encrypted. Your program will have to call decrypt again the next time it is run.
Instead, you have to create a new PDF and then copy content over from an existing document. The examples in this section will follow this general approach:. The write method takes a regular File object that has been opened in write-binary mode.
This allows you to combine multiple PDF files, cut unwanted pages, or reorder pages. Download meetingminutes. Enter the following into the interactive shell:.
Call PyPDF2. Call it again and pass it pdf2File to get a PdfFileReader object for meetingminutes2. These steps are done first for pdf1Reader and then again for pdf2Reader. You have now created a new PDF file that combines the pages from meetingminutes. PdfFileReader needs to be opened in read-binary mode by passing 'rb' as the second argument to open. PdfFileWriter needs to be opened in write-binary mode with 'wb'.
The pages of a PDF can also be rotated in degree increments with the rotateClockwise and rotateCounterClockwise methods. Pass one of the integers 90 , , or to these methods. Enter the following into the interactive shell, with the meetingminutes. We write a new PDF with the rotated page and save it as rotatedPage. The resulting PDF will have one page, rotated 90 degrees clockwise, as in Figure The return values from rotateClockwise and rotateCounterClockwise contain a lot of information that you can ignore.
Figure The rotatedPage. PyPDF2 can also overlay the contents of one page over another, which is useful for adding a logo, timestamp, or watermark to a page. Download watermark. Then enter the following into the interactive shell:. PdfFileReader open 'watermark. We then make a PdfFileReader object for watermark. The argument we pass to mergePage is a Page object for the first page of watermark.
Then we loop through the rest of the pages in meetingminutes. Finally, we open a new PDF called watermarkedCover. Figure shows the results. Our new PDF, watermarkedCover. PDFs can have a user password allowing you to view the PDF and an owner password allowing you to set permissions for printing, commenting, extracting text, and other features. The user password and owner password are the first and second arguments to encrypt , respectively.
If only one string argument is passed to encrypt , it will be used for both passwords. In this example, we copied the pages of meetingminutes. Before anyone can view encryptedminutes. You may want to delete the original, unencrypted meetingminutes. Even though there are lots of free programs for combining PDFs, many of them simply merge entire files together.
Call os. Write the output PDF to a file named allminutes. For this project, open a new file editor window and save it as combinePdfs. First, your program needs to get a list of all files with the. A trivial addition would be to use the names as the vertex labels and to color the vertices according to the gender.
Vertex labels are taken from the label attribute by default and vertex colors are determined by the color attribute, so we can simply create these attributes and re-plot the graph:. Note that we are simply re-using the previous layout object here, but we also specified that we need a smaller plot x pixels and a larger margin around the graph to fit the labels 20 pixels.
The result is:. Instead of specifying the visual properties as vertex and edge attributes, you can also give them as keyword arguments to plot :. This latter approach is preferred if you want to keep the properties of the visual representation of your graph separate from the graph itself.
To sum it all up: there are special vertex and edge properties that correspond to the visual representation of the graph. These attributes override the default settings of igraph see configuring-igraph for overriding the system-wide defaults.
Furthermore, appropriate keyword arguments supplied to plot override the visual properties provided by the vertex and edge attributes. The following two tables summarise the most frequently used visual attributes for vertices and edges, respectively:. The placement of the vertex label on the circle around the vertex.
This is an angle in radians, with zero belonging to the right side of the vertex. Shape of the vertex. Known shapes are: rectangle , circle , hidden , triangle-up , triangle-down. Several aliases are also accepted, see drawing. The curvature of the edge. Positive values correspond to edges curved in CCW direction, negative numbers correspond to edges curved in clockwise CW direction.
Zero represents straight edges. True is interpreted as 0. This is useful to make multiple edges visible. See also the autocurve keyword argument to plot.
These settings can be specified as keyword arguments to the plot function to control the overall appearance of the plot. Whether to determine the curvature of the edges automatically in graphs with multiple edges. The default is True for graphs with less than The bounding box of the plot. This must be a tuple containing the desired width and height of the plot. The default plot is pixels wide and pixels high.
The layout to be used. It can be an instance of Layout , a list of tuples containing X-Y coordinates, or the name of a layout algorithm. The default is auto , which selects a layout algorithm automatically based on the size and connectedness of the graph. The top, right, bottom and left margins of the plot in pixels. This argument must be a list or tuple and its elements will be re-used if you specify a list or tuple with less than four elements.
See the list of X11 color names in Wikipedia for the complete list. Alternatively you can see the keys of the igraph. This is a string according to one of the following formats where R , G and B denote the red, green and blue components, respectively :.
Example: " ff". RGB , components range from 0 to 15 in hexadecimal format. Example: " 08f". Example: , , 0 , [, , 0] or ", , 0". This can be done simply by passing the target filename as an additional argument after the graph itself. The preferred format is inferred from the extension. Yes doing that worked! Just one question : is the financial data accurate. Skip to content 05 May by Andrew Treadway.
May 6, at pm. Is there an equivalent library for google finance? May 7, at pm.
0コメント