Basic Concepts for BQN Policies
All IP data packets flowing through the BQN are assigned to a subscriber and to a flow. The BQN acts on traffic grouped per subscriber and per flow.
- A subscriber is an IPv4 address on the access side, or any IPv6 address from the same /64 subnet on the access side. See Subscriber Identification section for more details.
- A flow is a TCP connection, or a UDP flow, or a flow with another protocol (e.g. ICMP ping).A subscriber can have many flows at the same time.
To decide in a flexible way which functionalities to apply, the BQN uses policies. Policies define the actions to perform on the traffic (e.g., TCP Optimization or rate limitation), along with action parameters (e.g., a speed limit). There are three kinds of policies: flow policies, rate policies, and monitoring policies. Every flow is assigned a subscriber flow policy, while every subscriber is assigned a rate policy and a monitoring policy.
A data packet will receive all the functionalities specified in the flow policy for the flow it belongs to, and all the functionalities specified in the rate policy and the monitoring policy for the subscriber it belongs to. In this way, it is possible to specify different functionalities to all the flows of a subscriber, or just to certain flows of that subscriber.
Through flow policies we specify the following functionalities:
- TCP Optimization (TCPO).
- Limit the speed of all the subscriber flows matching an application definition (shaping). It is a per-subscriber limit, applying to the sum of the speed of all the subscriber flows assigned to the flow policy.
- Completely blocking of specific traffic types.
- Blocking only incoming connections from Internet of specific traffic types.
Through rate policies, we specify the following functionalities:
- Limit the total network speed of a subscriber.
Through monitoring policies, we specify the following functionalities:
- The amount of sampling when collecting DPI information for a subscriber.
The rate policy of a subscriber can be assigned from an external system (via RADIUS or via our REST API) or using profiles and rules. Flow policies and monitoring policies, are assigned using profiles and rules:
- Profiles classify the traffic according to some criteria (for example, an access profile identifies all the traffic from subscribers within a set of IP address ranges).
- Rules relate policies and profiles. For example, a rule may specify that some specific access profile is limited by a rate policy, i.e., that subscribers whose IP address are in some subnet will have a specific rate limit.
Profiles classify the traffic, and, when used by policy rules, determine the policy applied to a subscriber or a flow. There are different profile types, according to the properties being used for traffic classification. The current version supports the following profile types:
- Interface Profile: identifies the flows or subscribers whose first data packet comes in through a network interface listed by the profile.
- VLAN Profile: identifies the flows or subscribers whose first data packet uses a VLAN tag within the set of VLAN tags (or the absence of any tag) part of the profile.
- Policy Rate Profile: identifies the name of the subscriber rate policy. The profile may contain patterns with wildcards. For example, a policy rate profile containing pattern “premium-*” will match subscriber traffic with rate policy named “premium-gold” and “premium-platinum”.
- Internet Profile: identifies the flows involving an IP address on the Internet side, contained in the set of IP address ranges part of the profile. Optionally, Internet-side ports can also be specified (e.g., port 80).
- Access Profile: identifies the flows or subscribers involving an IP address on the Access side, contained in the set of IP address ranges part of the profile. Optionally, access-side ports can also be specified.
- Time Profile: defines time ranges. Optionally, ranges can be restricted to only some days of the week.
- Throughput Profile: identifies all the flows which were created while the total downlink traffic through the BQN was above the threshold specified by the profile.
- DPI (Deep Packet Inspection) Profile: identifies the flows that have a DPI domain that matches one of the domain patterns (signatures) part of the profile. There are a set of pre-defined DPI signatures, which include the signatures of popular applications (like the most important video-streaming apps or the most common software updates). See at the end of this section how to add custom signatures.
Profiles are configured in the menu option Configuration->Profiles.
An interface profile contains a list of network interfaces part of a data wire. It is true if the first packet is received by one of the interfaces of the profile. A network interface can only be part of one profile at the same time.
In the following example, a profile isdefined for one of the two wires of the BQN server and another profile for asecond wire:
A VLAN profile is a list of VLAN tags. The profile is true if the traffic has any of the VLAN tags defined by the profile.
The following example defines two profiles for two network areas and a third profile for traffic without VLAN tag.
Policy Rate Profile
This profile is used to select the Flow Policies based on the Rate Policy of the subscriber. It is a list of Subscriber Rate Policy names, or patterns with wildcards. Examples:
Internet and Access Profiles
Internet and Access profiles are a list of IP addresses. IP address ranges are also possible. An Internet profile defines a list of addresses on the Internet side (content sever addresses) and an Access profile defines addresses on the access side (subscribers addresses).
Optionally, it is possible to define a TCP or UDP port. To match any IP address and a given port, use 0.0.0.0/0 as the address range.
It is possible to load an IP address list from a text file. The text file format is one IP address or address range per line. To load a file, edit the profile and select, in the upper right menu, the option Replace Using File… (the profile mirrors the file address list) or the option Merge Using File… (the content of the file is added to the profile existing addresses).
Activates the rule during a period of time. A time profile is a list of time ranges, and it is true if any of the ranges is true. Ranges within the same profile cannot overlap.
A range can apply to all days of the week or just to certain days.
The following example shows:
- A rush hour profile at the end of any day (note how we define the 20:00 – 1:00 interval as two separate periods).
- A weekend profile during Saturday and Sundays.
- A time profile for working hours, with two ranges valid only from Monday to Friday
A throughput profile defines a threshold of the total downlink traffic through the BQN. It is true when the throughput is exceeded. The following example defines a threshold of 9 Gbps.
A DPI profile is a collection of signatures to detect in the information obtained through deep-packet-inspection. The signatures can have different types depending on the DPI information:
- HTTP-Host: the hostname in HTTP traffic
- HTTPS-SNI: the Service-Name-Identified in HTTPS traffic.
- QUIC-SNI: the Service-Name-Identified in QUIC traffic.
- QUIC-MVFST: presense of MVFST traffic (a QUIC variant).
- P2P-FILESHARING: presense of P2P traffic (BitTorrent).
- SPEEDTEST-OOKLA: presense of Speedtest traffic.
Signatures of the type HTTP-Host, HTTPS-SNI and QUIC-SNI must have a pattern indicating the expected content of the DPI information. Patterns can contain up to two wildchars. Examples: “prefix*”, “*suffix”, “*substring*”.
A set of pre-defined signatures can be loaded to the DPI profile selecting in the menu Add Predefined Signatures…
A DPI profile can be filled with custom signatures using the option Add Custom Signature….
Custom signatures can be loaded from a file using the option Add Signature File… A signature file is a text file with one line per signature, with the following format:
- <pattern> is a domain with wildcards (example: *domain-one.com)
- <pattern-type> is one of the following values: HTTPS-SNI, HTTP-host, QUIC-SNI
BQN identifies a subscriber as a distinctive IP address on the access side: one single IPv4 address or one IPv6 subnet. The default IPv6 subnet is /64 64 (to change it, go to Administration->General Settings and edit the field IPv6 prefix forsubscribers).
If there is a NAT between the BQN server and the real subscribers, subscribers whose IP addresses are translated to the same IP address would be considered as the same subscriber.
A new subscriber is identified when the first packet from an IP address is received. This is when the subscriber rate and monitoring rules are evaluated, to choose which policies to apply.
Subscriber Flow Policies
When a new flow is created, a subscriber flow policy is assigned to it, which specifies how to treat all the flows within that subscriber. The actions that can be defined in a subscriber flow policy are:
- TCP Optimization. Improves TCP traffic performance. It specifies whether to apply optimization to TCP traffic. It is recommended to enable it (the default value).
- Shaping per subscriber. It limits the combined speed of a subscriber flows to a given value. It is possible to limit in the downlink and/or uplink direction. The limit applies to all flows matching the policy belonging to the same subscriber. For example, if video streaming flows are assigned to a policy with a 6 Mbps subscriber limit, and the subscriber has three video streaming flows at the same time, the three flows will share the 6 Mbps limit (getting around 2 Mbps each). It is possible to define bursts that allow flows to exceed temporally the limit (see the endof this section).
- Shaping per flow. It limits the speed of one flow to a given value. It is possible to limit in thedownlink and/or uplink direction. The limit applies to any flow matching the policy. For example, if video streaming flows are assigned to a per flow 2 Mbps limit, a video flow cannot exceed those 2 Mbps. Shaping per flow can be combined with shaping per subscriber: for example, if there is a per subscriber 6 Mbps limit and a 2 Mbps per flow, a subscriber with four flows will have them limited to the 6 Mbps maximum (around 1.5 Mbps each). Per flow shaping has no burst option. Because per-flow shaping is not applied per subscriber, it can be used even when there is a NAT between the BQN and the end subscribers.
- Block. It blocks all flows falling in the blocking policy, in both directions, and does not let it proceed. It should be used with care, to avoid affecting traffic different to the one intended.
- Skip subscriber rate limitation. The traffic from flows getting this policy will no longer be affected by the rate limitation specified in the rate policy for this subscriber. They will only get the rate limitation specified by this flow policy (if any).
These policies are configured in the menu option Configuration->Subscriber Flows, selecting the tab POLICIES.
Shaping per subscriber
The following example defines a downlink speed limit of 10Mbps, an uplink limit of 8Mbps and bursts of 3 seconds of double the normal speed.
Regarding bursts, they are configured under Burst Options of the appropriate direction. Burst policy is defined by four parameters:
- Burst Rate: it is the maximum rate during the burst, typically bigger than the normal shaping max rate (e.g., allow burst of 20 Mbps for flows normally limited to 10Mbps).
- Burst Duration: it is the duration of the burst, for how long the burst rate can be sustained.
- Burst Threshold: it is an average speed that, if exceeded, prevent a new burst from happening. It is the way to control when a new burst can be granted. For example, for a 10Mbps limit with 20Mbps bursts, a 5Mbpsburst threshold will require the subscriber flows to drop the speed to half its normal limit before allowing a new burst.
- Burst Threshold Window: it is the period, in seconds, used to compute the average speed that is checked versus the threshold. The longer the window, the bigger the weight of past subscriber activity on the decision of grating a new burst.
Shaping per flow
The following example is a policy with a limit per flow of 4 Mbps in either direction.
It is possible to add a shaping per subscriber. Per flow and per subscriber shaping limits will act at the same time, per flow shaping limiting the speed of individual flows and subscriber shaping limiting the combined flow speed per subscriber.
Blocking incoming Traffic
It is possible to block incoming traffic, initiated from the Internet (TCP connections, UDP flows or other IP traffic like ICMP pings). Todo so, there are Drop Incoming Connections section as part of a Subscriber Flow policy.
Subscriber Rate Policies
Subscriber rate policies are applied per subscriber. The possible actions are:
- Maximum downlink speed. It is the maximum speed in the downlink direction for all traffic going towards the subscriber’s IP address.
- Maximum uplink speed. It is the maximum speed in the uplink direction for all traffic coming the subscriber’s IP address.
- Under Advanced Parameters, you can find the same burst options as for Subscriber Flow Policies.
- There is an Automatic Congestion Management (ACM) option, that will detect congestion and select a rate limit automatically (off by default).
These policies are configured in the menu option Configuration->Subscriber Rates, selecting the tab POLICIES.
See in section Subscriber Identification how all traffic from the same subscriber is identified.
Subscriber Rate Policies can also be created dynamically through the BQN external interfaces (RADIUS and REST API). In that case, the policy parameters, and the associations of the policy to the subscribers are controlled from the external interfaces and independent of BQN-configurated rules (configured rules are a fallback for subscribers without a policy assigned by the external interface). The external interfaces can assign subscribers both to rate policies defined though the external interface and to rate policies defined though the BQN GUI.
Automatic Congestion Management (ACM)
When the subscriber rate limits are unknown, the BQN can automatically detect them using machine learning so the BQN becomes the bandwidth management element, and the network can benefit from BQN reduced latencies. The ACM will also detect congestions below the subscriber rate limit, helping also when rate limits are known.
To enable this feature in BQN configured Rate Policies, enable the “Automatic Congestion Management" of a Subscriber Rate Policy (typically the rate-default one) :
To enable this feature in dynamic Rate Policies from RADIUS, enable the “Automatic Congestion Management" of Configuration->External Subscriber Data->RADIUS :
To enable ACM in dynamic Rate Policies created by the REST API, the external system must provide the congestion management field in the downlink direction when creating or updating the policy . Example:
Subscriber Monitoring Policy
Subscriber monitoring policies are applied per subscriber. They just determine the amount of sampling of DPI records for each subscriber. By default, the BQN comes with a monitor-default policy that applies to all subscribers, which automatically determines the number of DPI records (called UDRs, or Usage Data Records) to produce, so that the DPI statistics are meaningful and do not take too much CPU and disk to produce, and that is the recommended mode of operation:
It is possible to adjust the amount of UDRs for certain subscribers by creating a monitoring policy that specifies percentage of flows with UDRs (e.g. 0% or 100%). UDRs for 100% of flows will produce and accurate description of the subscriber usage statistics, but for performance reasons, it is recommended to set a 100% UDR monitoring policies only on a few subscribers at a time, because these policies can produce a huge number of records that could use all the available disk space and could also make the production of DPI statistics very slow and CPU-consuming.
Rules specify which policies to assign to each subscriber and flow, based on the profiles in the rule.
There are independent sets of rules for each policy type: subscriber flow rules select the appropriate subscriber flow policy for each flow, subscriber rate rules select the appropriate subscriber rate policy for each subscriber, and, similarly, subscriber monitoring rules select the appropriate subscriber monitoring policy for each subscriber.
A rule can use one profile of each type (or, alternatively, use the any option, if the profile type is indifferent), and it defines one and only one policy to apply.
Every set of rules may have many rules, but only the one with the best match will be selected for each flow or subscriber. To evaluate the rules in a way that maximizes performance, profiles are checked in order. This predefined order determines which rule is finally selected. A tree view of the rules helps in identifying which rule is selected in each case. See Decision Tree sections for details on the trees and the profile evaluation order.
Manually configured rule priorities are not used because of the performance penalties they entail and the burden on the operator to keep priorities consistent.
The subscriber flow rules are configured in the menu option Configuration-> Subscriber Flows, selecting the tab RULES TREE-VIEW or RULES TABLE-VIEW. Similarly, the subscriber rate rules are configurable in Configuration->Subscriber Rates and the subscriber monitoring rules in Configuration->Subscriber Monitoring.
Subscriber Flows Decision Tree
The evaluation of subscriber flow rules is as follows: when a new traffic flow is created (for example, a TCP connection), the profiles in the subscriber flow rule set are checked against that new flow, a matching rule selected, and with it the flow policy to apply.
For efficiency, profiles are evaluated in this predefined order:
The profile evaluation order defines a decision tree, whose nodes are the different profiles and with policies as leaves. The tree determines which rule is finally selected, because a rule can be excluded if it belongs to a branch that the decision tree does not follow. It may be the case that a flow matches more than one rule. In that case, the profile type order is important: for example, a rule matching the Interface profile would have priority over the rule matching the VLAN profile, and so on in the order specified above.
If two rules have a match with the same profile type, the more restrictive profile would have priority. For example, a flow from a subscriber with IP address 192.168.0.1 would match both an access profile with the 192.168.0.0/24 range and another access profile with the 192.168.0.0/16 range, but the first rule, with a narrower range (/24 vs/16), and therefore more restrictive, would be selected.
To facilitate the understanding of this order, the GUI includes a graphic representation of the decision tree, where the better matching path would lead to the selected policy (except when there is more than one match at the same profile level, when the most restrictive would win). It is accessible in Configuration->Subscriber Flows->Rules clicking the Tree View tab.
If there are common elements in two profiles of the same type and therefore a rule conflict, the decision tree will signal it so the rules can be reviewed by the operator and the conflict corrected. In the following example, two access profiles have an overlap (they both contain the same IP range). This will be signaled with a warning window and, also, one of the conflicting rules will have its number in yellow. Removing the overlap (for example making one of the profiles to have a more specific range), the conflict will disappear.
Subscriber Rate Decision Tree
The evaluation of the subscriber rate rules happens whenever a new subscriber session is detected (when traffic from a new access IP address is received). The profiles are checked against the subscriber session and a matching rule found, that will specify the rate policy to apply. For efficiency, the profiles are evaluated in the following pre-determined order:
Other profile types cannot be used in subscriber rate rules. For example, Internet profiles and DPI profiles will apply to some of the subscriber applications and not to others. Another example is throughput profile, because the rate rules are evaluated at the start of the subscriber session and the throughput changes continuously.
The decision tree is similar to the tree of subscriber flow rules. It is available in Configuration->Subscriber Rate->Rules selecting the tab RULES-TREE VIEW.
Subscriber Monitoring Decision Tree
The evaluation of the subscriber monitoring rules happens whenever a new subscriber session is detected (when traffic from a new access IP address is received). The profiles are checked against the subscriber session and a matching rule found, that will specify the rate policy to apply. For efficiency, the profiles are evaluated in the following pre-determined order:
Other profile types cannot be used in subscriber rate rules. For example, Internet profiles and DPI profiles will apply to someof the subscriber applications and not to others. Another example is throughput profile, because the rate rules are evaluated at the start of the subscriber session and the throughput changes continuously.
The decision tree is similar to the tree of subscriber flow rules. It is available in Configuration->Subscriber Monitoring->Rules selecting the tab RULES TREE-VIEW.
Check the Policy of a Subscriber and the Subscriber of a Policy
You can check the rate policies applied to a subscriber in Status->Subscribers. It lists the active subscribers, along with information about their flows and traffic volumes.
Clicking on the subscriber IP address provides detailed information, such as:
- Applied subscriber rate policy (Policy rate).
- Applied subscriber monitor policy (Policy monitor).
- Last measurement of downlink retransmissions in TCP traffic (Downlink current TCP RTX rate) and its average value (Downlink Average TCP RTX rate).
- Last measurement, in milliseconds of the access RTT (Downlink current RTT) and its minimum (Downlink min RTT).
You can dig in the active flows of a subscriber in Status->Flows->Per Subscriber, which shows the policy applied to each flow, among other things:
Likewise, given a policy, it is possible to see how many subscriber IP addresses are under each policy going to Status->Policies.
A click on a policy name leads to a list of subscribers using that policy (the ones with more volume consumption will be listed first).
The active subscribers, flows, etc. and displayed in a sortabletable. The first selector on the upper left defines how many entries to load in the table (10,000 by default). It is possible to adjust the number of entries shownper page (50 by default). To sort by a column, click on that column label. Clicking again will use reversed ordering.
When thousands of entries are loaded, a progress label will be displayed. There is the possibility of canceling the load. In that case, thetable will contain the entries loaded up to that point.
Several common examples of policies follow.
Implementing Subscriber Rate Plans
The objective is to apply the speed limits in each subscriber’s data plan.
The BQN applies these limits better than a conventional shaping element because, for TCP traffic (the most common),it does not need to discard packets. Furthermore, it uses independent queues per flow and that makes application latencies independent of each other, which greatly improves the experience of interactive applications. The following picture shows the queue structure, with a queue per flow and policy control at flow and subscriber levels.
The easiest way to implement Rate Plans is to use the RADIUS (see Radius section) or REST (see REST API section) interfaces. Rate policies will then be assigned by an external system for every subscriber. The external system can directly create those policies, or it can assign rate policies configured from the BQN GUI.
If an external system cannot be used, you can simply create one rate policy for that plan, an access policy with all the subscriber IP addresses (or ranges) assigned to every plan, and then one rule for each plan, linking the corresponding access profiles and rate policies. The following is an example of the rule tree:
It is also possible to define such rules just for one test IP address that is used during a proof-of-concept to see the BQN performance as a bandwidth manager:
Limiting the Speed of Some Applications
The goal could be to reduce the network peak throughput to mitigate the congestion at the peak hour. To that end, a DPI profile is defined (video in the example) to identify the applications to limit. This example makes use of video streaming predefined signatures. To include them, in Add DPI profile, select Add Predefined Signatures and choose the Video Streaming predefined signature.
Also, a throughput profile is created with the traffic load from which to start limiting (above-5Gbps in this example). Then, a subscriber flow policy (flow-10Mbps in the example) is created with a downlink limit (Downlink shaping per Subscriber) set at 10 Mbps. Finally, the DPI profile, the throughput profile and the subscriber flow policy are tied together in a subscriber flow rule.
Limiting the Speed of Some Applications with NAT
The goal is to reduce the network peak throughput to mitigate the congestion at the peak hour. In this scenario there is a NAT between the BQN server and the end subscribers and, therefore, a shaping per subscriber is not possible.
We will use the same Throughput and DPI profiles of the previous example, but we will use a per-flow shaping policy.
Services not limited by the Subscriber Rate
The goal is to preserve the quality of experience of some services by granting throughput to them even when the subscriber rate plan is fully used. An example could be a Voice over IP(VoIP) service hosted by the ISP. The policy is limited to subscribers with gold plans. It is created with an Internet profile (voip) with the IP address and port of the ISP-hosted VoIP service, a policy-rate profile and a flow policy (with Skip subscriber rate limitation selected to On). Next, the Internet profile and the policy-rate profiles are linked to the policy by a subscriber flow rule.
In this scenario, some applications need blocking, for example, servers that are sources of attacks. To do this, an Internet profile is created (malicious-apps in the example) to identify the IP addresses to block. Next, a subscriber flow policy is defined with block action (with name flow-block in this example) and, finally, the Internet profile and the subscriber flow policy are combined in a subscriber flow rule.
Exclude Traffic from TCP Optimization
The objective is that the BQN will not optimize certain traffic. For example, some subscribers. To that end, an access profile is defined (subs-no-tcpo in the example), with the subscriber IP addresses to exclude. Next, a subscriber flow policy is defined with optimization set to off (flow-no-tcpo in the example) and, then the access profile and the subscriber flow policy are combined in a subscriber flow rule.
This setup is equivalent to the IP address blacklist in BQN Release 3 software.
Another example would be to use an Internet profile to exclude some applications per TCP port.